Lecture 13 Page 1 CS 239, Spring 2007 Data Presentation CS 239 Experimental Methodologies for System Software Peter Reiher May 22, 2007

Lecture 13Page 1CS 239, Spring 2007

Data Presentation CS 239

Experimental Methodologies for System Software

Peter ReiherMay 22, 2007


Outline

• General issues of data presentation

• Common methods of deceiving with data

• Graphical data presentation


Data Presentation

• Experiments are usually not done just for your own edification

• Generally, you need to describe the results to others

• How do you effectively present your data?


The Key Point

• You aren’t presenting numbers or graphs• You are presenting insight

– Backed up by solid evidence• Think about what you want your audience

to learn first• Worry about your numbers and graphs later

– And choose numbers and graphs that explain and defend your insights


A Corollary

• Present as little data as you need to present– But no less

• Don’t print every possible graph• Don’t show redundant graphs that simply

repeat the same point• Make sure each piece of data you present:

– Says something important– Says something the others don’t


Another Consideration

• Know your audience• Is it likely to believe your results?• Will it be skeptical?• Does it have the expertise to

understand complex issues?• Tailor your presentation to the

audience


Methods of Presenting Data

• In written form

• In live presentations


Presenting Data in Written Form

• Simple in-line description– “System x was 12% faster”– Only good for short, simple statements– Especially in summaries/conclusions

• Tables – Useful if seeing individual numbers is helpful– Also good if reader will make many comparisons across table

entries– But big tables are hard to read– So minimize number of entries– Avoid big tables with lots of rows and columns

• Graphics– Often best way to explain large, complex data sets


Presenting Data in Talks

• If it’s important, make sure it’s written down– On your slides, handouts, or

somewhere else they can go to later• Generally want it short and snappy

– Tables usually not appropriate– Brief mentions and graphics better


What About the Full Data?

• You should always save all the data• When possible, make it available to everyone

– Not possible if sensitive– Or legally prohibited– Or potential liability– Or a company asset

• Otherwise, put it in a suitable repository• Or put it on the web yourself• Make sure you include information on data’s

format and basic meaning


Using Data Deceptively

• Well-known techniques for using real data in deceptive ways

• Often called “games”

– Benchmarking games

– Ratio games


Benchmarking Games

• Techniques for producing data likely to be deceptive

• Methods of designing experiments that lead to results that are “advantageous”– Assuming experimenter cares how

experiment turns out– As he often does

• Not always intentional


Differing Configurations

• Use different configurations for the same workload on two systems

• E.g., a Linux vs. Windows comparison where one system has less memory

• Be suspicious when all conditions aren’t fully mentioned– Particularly if data comes from different

sources


Differing Compilation Options

• One version of the code is compiled with optimization

• The other isn’t

• True performance difference, but . . .

• Again, if data gathered from different sources, might be hard to tell


Biased Testing Specifications

• Experiment designed to favor situations good for one alternative

• E.g., a system that makes heavy use of disk is tested on a machine with a fast, expensive drive

• Or specifying a 10 Mbps incoming link for testing a firewall that runs poorly above 11 Mbps


Playing With the Workload

• Design workload so it fits known cycles of one system

• E.g., design workload to hide page faults

• Or choose arbitrary workload that isn’t representative– But does well on preferred system


Very Small Benchmarks

• Set benchmark sizes small to hide scaling effects

• E.g., hide poor disk performance by working out of cache

• Or conceal poor scaling of key data structure by limiting number of entries in workload


Tweaking Benchmarks

• Can alter aspects of benchmarks specifically to play to system strengths

• E.g., removing all images from web benchmark if system handles them poorly

• Increasing proportions of graphics operations in a benchmark to favor machine with special graphics hardware


• Ratios offer special possibility to conceal information

• Sometimes used specifically to hide what you don’t want to show

• How do you do that?

• How do you tell when others do that?

Ratio Games


Choosing a Base System

• A simple game method using ratios

• Run workloads on two systems

• Normalize performance to system you want to look good

• Take average of ratios

• Presto: you control what’s best


Example of Choosinga Base System

• A processor comparison (lower is better)

• Raw Data:

System

Benchmark A B

Block 41.16 51.5

Sieve 63.17 48.08

Sum 104.33 99.58

Average 52.17 49.79


Making System A Look Better

• Calculate ratio with A as base

System

Benchmark A B

Block 1.00 1.25

Sieve 1.00 0.76

Sum 2.00 2.01

Average 1.00 1.01


Making System B Look Better

• Use B as a base

System

Benchmark A B

Block .80 1.00

Sieve 1.31 1.00

Sum 2.11 2.00

Average 1.06 1.00


Lying With Ratio Metrics

• Pick a metric that is itself a ratio

– E.g., power = throughput response time

– Or cost/performance

• Handy because division is “hidden”


Relative Performance Enhancement

• Compare systems with incomparable bases

• Turn into ratios

• Example: compare Ficus 1 vs. 2 replicas with UFS vs. NFS (1 run on chosen day):

• “Proves” adding Ficus replica costs less than going from UFS to NFS

"cp" Time RatioFicus 1 vs. 2 197.4 246.6 1.25UFS vs. NFS 178.7 238.3 1.33


Ratio Games with Percentages

• Percentages are inherently ratios– But disguised– So they’re great for ratio games

• Example: Passing tests

• A is worse, but looks better in total line!

Test A Runs A Passes A % B Runs B Passes B %1 300 60 20 32 8 252 50 2 4 500 40 8

Total 350 62 18 532 48 9


More on Percentages

• Psychological impact– 1000% sounds bigger than 10-fold (or 11-fold)– Great when both original and final performance

are lousy• E.g., salary went from $40 to $80 per week

• Base should be initial, not final value– E.g., price can’t drop 400%


Strategies for Winninga Ratio Game

• Can you win?

• How to win


Can You Winthe Ratio Game?

• If one system is better by all measures, a ratio game won’t work– But recall percent-passes example– And selecting the base lets you change the

magnitude of the difference• If each system wins on some measures, ratio

games might be possible (but no promises)– May have to try all bases


How to WinYour Ratio Game

• For LB metrics, use your system as the base• For HB metrics, use the other as a base• If possible, adjust lengths of benchmarks

– Elongate when your system performs best– Short when your system is worst– This gives greater weight to your strengths


Correct Analysis of Ratios

• Discussed in detail in previous lectures

• Generally, harmonic or geometric means are appropriate

– Or use only the raw data

• If someone’s doing something else with ratios, be suspicious


• Reference works

• Types of variables

• Guidelines for good graphics charts

• Common mistakes in graphics

• Pictorial games

• Special-purpose charts

Graphical Data Presentation


• Edward R. Tufte, The Visual Display of Quantitative Information, Graphics Press, Cheshire, Connecticut, 1983.

• Edward R. Tufte, Envisioning Information, Graphics Press, Cheshire, Connecticut, 1990.

• Darrell Huff, How to Lie With Statistics, W.W. Norton & Co., New York, 1954

Useful References Works


• Qualitative

– Ordered (e.g., modem, Ethernet, satellite)

– Unordered (e.g., CS, math, literature)

• Quantitative

– Discrete (e.g., number of terminals)

– Continuous (e.g., time)

Types of Variables


Charting Basedon Variable Types

• Qualitative variables usually work best with bar charts or Kiviat graphs

– If ordered, use bar charts to show order

• Quantitative variables work well in X-Y graphs

– Use points if discrete, lines if continuous

– Bar charts sometimes work well for discrete


• Principles of graphical excellence• Principles of good graphics• Specific hints for specific situations• Aesthetics• Friendliness• Much of this based on Tufte’s

principles of graphics design

Guidelines for Good Graphics Presentation


Principlesof Graphical Excellence

• Graphical excellence is the well-designed presentation of interesting data:

– Substance

– Statistics

– Design


Graphical Excellence (2)

• Complex ideas get communicated with:

– Clarity

– Precision

– Efficiency


Graphical Excellence (3)

• Viewer gets:

– Greatest number of ideas

– In the shortest time

– With the least ink

– In the smallest space

• Requires telling truth about data


Principles of Good Graphics

• Above all else show the data

• Maximize the data-ink ratio

• Erase non-data ink

• Erase redundant data ink

• Revise and edit


An Important Caveat

“The principles should not be applied rigidly or in a peevish spirit; they are not logical or mathematically certain; and it is better to violate any principle than to place graceless or inelegant marks on paper.”

Edward Tufte, “The Visual Display of Quantitative Information”


Above All ElseShow the Data

y = 1E-05x + 1.3641

R2 = 0.0033

0

1

2

3

4

5

0 5000 10000 15000File size (bytes)

Tim

e to

fet

ch (

seco

nds) Linear model

Where’s the data?


Above All ElseShow the Data

y = 1E-05x + 1.3641

R2 = 0.0033

0

1

2

3

4

5

0 5000 10000 15000File size (bytes)

Tim

e to

fet

ch (

seco

nds) Linear model

Limitations of linear model much clearer now


Maximize theData-Ink Ratio

1st Qtr

3rd Qtr

010203040506070

80

90

East

West

North


Maximize theData-Ink Ratio

0

20

40

60

80

100

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

East West

North

Important features become much clearer


Erase Non-Data Ink

05

101520253035404550556065707580859095


East

West

North

Note disconcerting vibration effects


Erase Non-Data Ink

0

20

40

60

80


Eas

t

Wes

t

Nor

th


Erase Redundant Data Ink

20.4

27.4

90

20.4

38.634.6

31.6

46.9 45 43.9

30.6

45.9

0

20

40

60

80


Eas

t

Wes

t

Nor

th


Erase Redundant Data Ink

0

20

40

60

80


Eas

t

Wes

t

Nor

th


Revise and Edit

0

20

40

60

80

100


Default Microsoft Powerpoint Chart

East

West

North


Revise and Edit

Remove Decorative Effects

0

20

40

60

80

100


East

West

North


Revise and Edit

Remove Clutter

0

20

40

60

80

100


East

West

North


Revise and Edit

Make Legend Simple to Interpret

0

20

40

60

80

100


East

West

North


Revise and Edit

Eliminate Superfluous Ink

0

20

40

60

80

100


East

West

North


Revise and Edit

Eliminate Red/Green Distinctions

0

20

40

60

80

100


East

West

North


Revise and Edit

Choose Better Fonts

0

20

40

60

80

100


East

West

North


Specific Things to Do

• Give information the reader needs

• Limit complexity and confusion

• Have a point

• Show statistics graphically

• Don’t always use graphics

• Discuss it in the text


Give Informationthe Reader Needs

• Show informative axes

– Use axes to indicate range

• Label things fully and intelligently

• Highlight important points on the graph


Giving Informationthe Reader Needs

0

20

40

60

80

1 2 3 4

E

W

N

What does any of this mean?


Giving Informationthe Reader Needs

0

20

40

60

80


Salesin

Millions

MicrosoftContractSigned

East

North

West


Limit Complexityand Confusion

• Not too many curves

• Single scale for all curves

• No “extra” curves

• No pointless decoration (“ducks”)


0

10

20

30

40

50

60


0

20

40

60

80

100

120 West

North

Northeast

Southwest

Mexico

Europe

Japan

East

South

International

Limiting Complexityand Confusion


International Sales

0

20

40

60

80

100

120


Millionsof

Dollars

Japan

Mexico

Europe

Limiting Complexityand Confusion


Have a Point

• Graphs should add information not otherwise available to reader

• Don’t plot data just because you collected it

• Know what you’re trying to show, and make sure the graph shows it


Having a Point

• Sales were up 15% this quarter:

0

20

40

60

80

100

120

140

1st Qtr 2nd Qtr


Having a Point

User Time of Copy Benchmarks

0

0.01

0.02

0.03

0.04

1 Replica 2 Replicas 3 Replicas 4 Replicas

cp rcp


Having a Point

0

500000

1000000

1500000

2000000

2500000

3000000

3500000

4000000

4500000

Modem Ethernet ATM Satellite

Throughput

Latency


Having a Point

1

10

100

1000

10000

100000

1000000

0.01 0.1 1 10 100 1000

Throughput (Kbits/sec)

Latency(m s) Ethernet

Modem

ATM

Satellite


Show Statistics Graphically• Put bars in a reasonable order

– Geographical

– Best to worst

– Even alphabetic

• Make bar widths reflect interval widths

– Hard to do with most graphing software

• Show confidence intervals on the graph

– Examples will be shown later


Don’t Always Use Graphics

• Tables are best for small sets of numbers

– Tufte says 20 or fewer

• Also best for certain arrangements of data

– E.g., 10 graphs of 3 points each

• Sometimes a simple sentence will do

• Always ask whether the chart is the best way to present the information

– And whether it brings out your message


Discuss It in the Text

• Figures should be self-explanatory– Many people scan papers, just look at graphs– Good graphs build interest, “hook” readers

• But text should highlight and aid figures– Tell readers when to look at figures– Point out what figure is telling them– Expand on what figure has to say


Aesthetics

• Not everyone is an artist

– But figures should be visually pleasing

• Elegance is found in

– Simplicity of design

– Complexity of data


Principles of Aesthetics

• Use appropriate format and design• Use words, numbers, drawings together• Reflect balance, proportion, relevant scale• Keep detail and complexity accessible• Have a story about the data (narrative quality)• Do a professional job of drawing• Avoid decoration and chartjunk


Use AppropriateFormat and Design

• Don’t automatically draw a graph

– We’ve covered this before

• Choose graphical format carefully

• Sometimes a “text graphic” works best

– Use text placement to communicate numbers

– Very close to being a table


GNP: +3.8 IPG: +5.8 CPI: +7.7 Profits: +13.3

CEA: +4.7

DR: +4.5

NABE: +4.5

WEF: +4.5

CBO: +4.4

CB: +4.2

IBM: +4.1

CE: +2.9

NABE: +6.2

IBM: +5.9

CB: +5.5

DR: +5.2

WEF: +4.8

IBM: +6.6

NABE: +6.5

CB: +6.2

WEF: +21

DR: +10.5

IBM: +10.4

CE: +6.5

WEF: 6.8

CB: 6.7

NABE: 6.7

IBM: 6.6

DR: 6.5

CBO: 6.3

CEA: 6.3

Unempl: 6.0

About a year ago, eight forecasters were asked for

their predictions on some key economic indicators.

Here’s how the forecasts stack up against the

probable 1978 results (shown in the black panel).

(New York Times,

Jan. 2, 1979)

Using Text as a Graphic


Choosinga Graphical Format

• Many options, more being invented all the time

– Examples will be given later

– See Jain for some commonly useful ones

– Tufte shows ways to get creative

• Choose a format that reflects your data

– Or that helps you analyze it yourself


Use Words, Numbers, Drawings Together

• Put graphics near or in text that discusses them

– Even if you have to murder your word processor

• Integrate text into graphics

• Tufte: “Data graphics are paragraphs about data and should be treated as such”


Reflect Balance, Proportion, Relevant Scale

• Much of this boils down to “artistic sense”

• Make sure things are big enough to read

– Tiny type is OK only for young people!

• Keep lines thin

– But use heavier lines to indicate important information

• Keep horizontal larger than vertical

– About 50% larger works well


Poor Balanceand Proportion

• Sales in the North and West districts were steady through all quarters

• North sales varied widely, significantly outperforming the other districts in the third quarter 0

10

20

30

40

50

60

70

80

90

100



Better Proportion

• Sales in the North and West districts were steady through all quarters

• North sales varied widely, significantly outperforming the other districts in the third quarter

0

50

100

Q1 Q2 Q3 Q4


Keep Detail and Complexity Accessible

• Make your graphics friendly:

– Avoid abbreviations and encodings

– Run words left-to-right

– Explain data with little messages

– Label graphic, don’t use elaborate shadings and a complex legend

– Avoid red/green distinctions

– Use clean, serif fonts in mixed case


An Unfriendly Graph

050

100150200250300350400450

1 REPL 2 3 4 5 6 7 8

Time

CP

FIND

FINDGREP

GREP

LS

MAB

RCP

RM


A Friendly Version

0

100

200

300

400

1 2 3 4 5 6 7 8

Number of Replicas

Time in Seconds

Copy

Compile

Remove

Note almost no growth incompile/remove times


Even Friendlier

0

100

200

300

400

Copy Compile Remove

Benchmark and Number of Replicas

Time in Seconds

Note slower growth incompile and remove times

1 Replica

8 Replicas(note departurefrom linearity)


Have a Story About the Data (Narrative Quality)

• May be difficult in technical papers• But think about why you are drawing• Example:

– Performance is controlled by network speed– But it tops out at the high end– And that’s because we hit a CPU bottleneck


Showing a StoryAbout the Data

0

20

40

60

0 2 4 6 8 10 12

Network Bandwidth (Mbps)

Transactionsper

SecondCPU bottleneck

reached


Do a Professional Jobof Drawing

• This is easy with modern tools

– But take the time to do it right

• Align things carefully

• Check the final version in the format you will use

– I.e., print the PDF one last time before submission

– Or look at your slides on the projection screen

• From the machine you’ll use

• Preferably with the projector you’ll use


Avoid Decorationand Chartjunk

• Powerpoint, etc. make chartjunk easy• Avoid clip art, automatic backgrounds, etc.• Remember: the data is the story

– Statistics aren’t boring– Uninterested readers aren’t drawn by cartoons– Interested readers are distracted

• Does removing it change the message?– If not, leave it out


Examples of Chartjunk

0

10

20

30

40

50

60

70

80

90


Gridlines!Vibration

Pointless

Fake 3-D Effects

Filled “Floor” Clip Art

In or out?

Filled

“Walls”

Borders and

Fills Galore

Unintentional

Heavy Lines

Filled Labels

Documents

Lecture 13 Page 1 CS 239, Spring 2007 Data Presentation CS 239 Experimental Methodologies for System Software Peter Reiher May 22, 2007