No Silver Bullet? Silver Buckshot May Work · 2013-08-25 · "No Silver Bullet? Silver Buckshot May Work" Presented by: Gregory Pope Lawrence Livermore National Laboratory Brought

KT3 Keynote 11/10/2011 12:45:00 PM

"No Silver Bullet? Silver Buckshot May Work"

Presented by:

Gregory Pope Lawrence Livermore National Laboratory

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

Gregory Pope Lawrence Livermore National Laboratory

With more than forty years of experience developing software in the commercial and

government sectors, Gregory Pope currently works for the Lawrence Livermore

National Laboratory as a software quality engineering group leader, and verification and validation

project leader for advanced simulation. Previously, Greg founded and ran a software testing

company, patented automated software testing tools, and held management and technical positions

involving mission-critical testing of military systems and development of software code for avionics

and aerospace uses. Greg has given industry keynote addresses, written technical papers, taught on

software quality internationally, and been a consultant.

9/28/2011

1

Lawrence Livermore National Laboratory

Gregory PopeSQE Group Leader

No Silver Bullet? Silver Buckshot May Work

This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344

2LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory

Silver Buckshot

� The proverbial Silver Bullet

� Most common request

� Definition of Better

� Good ideas reincarnate

� Some modern challenges

� Buckshot – Common problems and solutions

9/28/2011

2


No Silver Bullet

The phrase typically appears with an expectation that some new technological development or practice will easily cure a major prevailing problem.


Example Silver Bullets

High Level Language -------------------------------------- c. 1968

Top Down Structured Programming --------------------c. 1974

Waterfall Lifecycle ------------------------------------------ c. 1976

DoD Standard 2167A ------------------------------------- c. 1988

Computer Aided Software Engineering (CASE)----- c. 1990

Object Oriented Design and Programming ---------- c. 1992

CMM/CMMI -------------------------------------------------- c. 1993

Rational Unified Process --------------------------------- c. 1996

Automated Testing Tools ---------------------------------- c. 1995

Continuous Integration ------------------------------------ c. 1999

9/28/2011

3


Each Helped Make things Better

Increased productivity

Less spaghetti code

In process error detection, iterative, incremental spins

More documentation is not the answer - RUP, artifacts

Frameworks and plug ins - Eclipse, Team Forge

Reusability - Java, Python, Ruby, C#

Regulated vendor minimum competencies, Outsource

Expect requirements to change - Agile

Nightly regression testing – Junit, Subversion, QTP

Defects detected earlier – CMAKE, BuildBot, Jenkins


Common Question

What can we do to make our process better?

9/28/2011

4


Better for Who

A. Make the software better -------------------- c. 1970

B. Make the user experience better------------c. 1990

C. Make the developer experience better-----c. 2005

Better = A & B & C


Better

� If the improvements do not make the developers life better, they will probably not be easily adopted. Dah

� Examples:

• Integrated Development Environments (IDEs)

• Networking

• Static analysis

• Continuous integration

• Distributed Code repositories

9/28/2011

5


Most Common Process Improvement Response….

� Good idea, but we are busy, we will do it later.


An Observation: Later

� The “land of later” is a mythical place where nothing ever happens.

� Why?

• Because if you are successful you will be whisked off to a new project.

• If you unsuccessful you will be whisked off.

9/28/2011

6


Bettering Unit Testing

� Debug and try it

� Write unit test drivers and stubs manually

� Stand alone framework to write unit tests

� Automated tool (Automated Regression Tests to run

periodically)

� Automated the Automated tool, run unit tests when a code

change is detected


Bettering Coding Style

� Leave coding styles up to individuals

� Ask team to follow a coding standard

� Have a plug in tool that formats the code to the correct

style (i.e. indents) in the IDE

� Have a static analyzer that checks for style violations

9/28/2011

7


Bettering Peer Reviews

� No peer review of code

� Meeting to peer review code

� Meeting to discuss findings in code

� Collaborative tool to allow code review

� Collaborative tool with built-in diff and tracker interface

(to requirements and bugs) and CM tool


Bettering Static Analysis

� Compiler, debugging, and testing to find bugs

� Static analysis on integrated code

� New rules added to reduce false alarms

� Static analysis on code as it is built

� Automated static analysis on check in and nightly

� Automate the automated static analysis emailing only new issues found

9/28/2011

8


Is Better Cheaper and Faster?

� Cost to manually test – 4 hours per bug*

� Cost of automated test – 1 hour per bug*

� Cost of static analysis – 10 minutes per bug**

* Cost to design tests (scripts) and execute

** Cost to build and triage code

Source: William Oliver LLNL “Quantifying the Value of Static Analysis”, Starwest 2011


Source: Boris Beizer Software System Testing and Quality Assurance, page 34

All bugs are not structuralLLBut some structural bugs cause system, functional, and data bugs.

Static Analysis

Structural Bugs 27%

Functional Bugs 27%

15%

System16%

Data, Code, Other30%

15%

15%

9/28/2011

9


Hawthorne Effect

� Study from 1927 to 1932 at a Western Electric Company Plant in Cicero, Illinois by Harvard Researcher Elton Mayo.

� One reasonable conclusion is that the workers were pleased to receive attention from the researchers who expressed an interest in them.

� Any new tool or process can cause process improvement.


Best Ideas Seem to get Reincarnated

� Built-in-test for avionics ------ c. 1972

� Design by Contract

� Assertions

9/28/2011

10



� Mission critical software --- c. 1972

� Test Driven Design

� Pairs Programming

Missile Simulator Software

New Software

RWR Software



� ITT 1978 “stand up” meetings

� Accomplished, Plan to Accomplish, Obstacles

� Five minutes in length, can not sit down

� All decision makers present

� Scrum meeting

� “Main purpose of SCRUM is to find problems early”.

� Assumption is if a problem is identified early it will get fixed early. Not for all however LLL

Source: Lunch with Ken Schwaber 2007

9/28/2011

11


SCRUM

The XYZ project is

ahead of schedule

and under budget.

What do you mean

the software isn't

ready for test yet?

OK. You've got to

finish. Give it to test

for a couple of days.

We have got to add

a month to this

schedule.

Admits there is

a problem.

We should have time

to test if we get the

software on schedule.

You can't use our

test schedule to keep

writing new code.

We can't even begin

to test XYZ, let alone

finish, in two days.

You can't deliver XYZ

without a minimum of

testing.

Admits the need to

automate testing.

Let's plan a picnic to

celebrate the success-

ful XYZ delivery.

Perhaps we should

delay the picnic by a

week or so.

Perhaps we should

delay the picnic by

a month.

Refers Test Manager

to Employee

Assistance Program.

Finds job for old

program manager.

Admits self to

Employee Assistance

Program.

Trust me. We just

have a few minor

functions to add.

That last function

required changing 50%

of our design. Sorry.

Tomorrow, for certain,

will be the day we

give it to test.

We didn't know you

wanted all functions

in the first version.

Admits the need to

plan based on prior

experiences.

XYZ will increase our

sales by 50% next

quarter.

XYZ will be even

better than we

first thought.

XYZ is so good, test

hasn't even found any

bugs in it.

What do you mean

it's late and doesn't

work? We'll lose

market share!

Admits the need to keep

in closer contact with

engineering.

Denial

Anger

Depression

Bargaining

Acceptance

XYZ PROGRAM

MANAGER

TEST-QA

MANAGER

HR

MANAGER

ENGINEERING

MANAGER

MARKETING

MANAGER

DEPT

STAGE

Get to acceptance faster LL


Contemporary Challenges

� Prototypes become products, options:

• Start over

• Refactor

• Wrappers

−Disable features

− Input checking

−Disable inputs

• Testing in Domain of Interest

9/28/2011

12



� Code Security

• Static Analysis

• Security Testing

• C, C++ written before there was an internet *

• Security not a function or test, it is an intrinsic value*

� Optimistic Schedules

• Estimation Tools

• Acceleration

• Release Prediction Tools

* Source: James Gosling, inventor Java, speaking at Sandia Labs 8/18/2011



� More Code Writers

• Automated workflow

� Text Books

• Some Examples

� Understanding Requirements

• Some thoughts

� Good Design

• Some thoughts

9/28/2011

13



� Buggy Text Book Code

Type first integer : 5Type second integer : 6Type third integer : 7

5 is in the range 30 to 199 : FALSEat least two integers you typed are equal : FALSE5 != 6 : TRUENOT (5 < 6) : FALSE5 <= 6 : TRUE7 > 6 : FALSE(7 = 5) AND (6 != 7) : FALSE(7 <= 5) XOR (6 >= 7) : FALSE(7 > 5) AND (6 <= 7) : TRUE

Polarity


More Buggy Text Book Code

Please enter January 1's starting day;

A 0 indicates January 1 is on a Monday,A 1 indicates January 1 is on a Tuesday,

etc: 5

Enter the year you want the calendergenerated: 2011

The calendar for the year 2011

January Sun Mon Tue Wed Thu Fri Sat--- --- --- --- --- --- ---

1 23 4 5 6 7 8 9

10 11 12 13 14 15 1617 18 19 20 21 22 2324 25 26 27 28 29 3031

November Sun Mon Tue Wed Thu Fri Sat--- --- --- --- --- --- ---

1 2 3 4 5 67 8 9 10 11 12 13

14 15 16 17 18 19 2021 22 23 24 25 26 2728 29 30

Change string to Sunday and Monday

Spelling

9/28/2011

14


Tracking Tool: - Collabnet


Requirements

� The real answers are 6 levels down

� Who

� What

� Where

� Why

� How

� Who

� Dynamic tracing to code and tests

9/28/2011

15


Non-Functional Requirements

� Usability

� Maintainability

� Scalability

� Availability

� Extensibility

� Security

� Portability

� Safety


Requirements

� Prototypes help elicit requirements

� Good question asking skills

� Experience

� Can be too many or too few

� Requirement writers make good test designers.

9/28/2011

16


Getting Requirements

� Never and Always LLLLLLLLL...

� Talk to the help desk folks

� Testing not needed for reuse (fail)

� Sit in the user’s seat


Emerging Chaos

Chaos

Perceived Order

Emerging Problem Solving

Chaos

Emerging Order

Order

Deliberate Problem Seeking

Instead of trying to make order exist initially,chaos is allowed to exist so all stakeholders can problem seek together.

Emerging chaos closely resembles the waterfallor V-model, an attempt to capture all (or most) of the software requirements at the outset (create apparent order).

Source: Moshe F. Rubinstein and Iris R. Firstenberg, The Minding Organization, Bring the Future to the Present and

Turn Creative Ideas into Business Solutions, University of California, Los Angeles, John Wiley and Sons, pg. 91

Requirement Chaos for Creativity

9/28/2011

17


Software Design

� Hard to get developers to do this

� Peer Pressure

� Manager Pressure

� Director Pressure

� Owner Pressure

� Even when done the design is seldom updated


Code is the Design

� The source code is an abstraction of the executing binaries.

� It is a detailed design

� Source Code is like a blueprint is to a structure

� Architecture Design is important, like a scale model is to a structure.

9/28/2011

18


Safety Moment

� Most safety problems with software intensive systems did not involve the software failing.

� Most safety problems were the interaction of the software components with other system components.

� Reliability is not the same as safety in software.

� Software does not fail the same as hardware.

Source: Nancy Levenson, Safeware System Safety and Computers


Security Check:

� Weak coding practices allow hackers to exploit code

� Examples:

• Array bounds overflow

• Memory Leaks

• Tainted Inputs

• Null Pointer Dereference

• Uninitialized Variables

• Use tools to find this stuff

� So Better supports more Secure

9/28/2011

19


Software Management

� Graded approach to rigor

� In general: The amount of rigor used for software process is proportional to the severity of the consequence of failure of the software being built.

� Managed, Documented, Understood


Deja Vu All Over Again

� Running out of memory again:

Power consumption for an exascale (1018 flops) system in 2018 even under optimistic assumptions would be 100–200 MW, $100 Million annual electric bill.

� Memory reliability again

Cosmic radiation has a much higher probability of disturbing RAM memory.

Title Source: Yogi Berra, New York Yankees

http://www.lbl.gov/CS/html/SC08ExascalePowerWorkshop/gara.pdfFlash Memory Reliability NEPP 2008 Task Final Report, Yuan Chen, Jet Propulsion Laboratory, Pasadena, California

9/28/2011

20


Adoption of Better By Decree

Is Difficult

� Generational

� Cultural

� Industrial

� Geographical

� Language


Generational Factors

Matures1933-1945

Boomers1946-1964

Gen X1965-1976

Gen Y1977-1998

34 Million 76 Million 41 Million 75 Million

Source: Love’Em or Lose’Em, Kay and Jordan-Evans, page 236

9/28/2011

21


Generational Cheat Sheet

Matures Boomers Generation X Generation Y

Work EthicWork 'til you

drop.

Work long hours

and tell your

manager about

it.

Personal life first,

work is

important.

Lifestyle comes first.

Loyality Loyal to employer.

Loyal to

employer, with

reservations.

Career and

professional

loyalty.

Career options.

TechnologyTechnology

fascination.

Technology

challanged.

Technology

proficient.Technology immersed.

Reporting

Relationships

Strong chain of

command.

Chain of

command.

What is the

purpose of a

chain of

command?

Be respectful but

move ahead.

Source: Love’Em or Lose’Em, Kay and Jordan-Evans, page 245


Cultural Differences

� Globalization makes this more critical

� Teaching in Asia

• Questions written

• Prescriptive approach

� Teaching in Israel

• Highly Interactive

• Principle based approach

� Teaching in California

9/28/2011

22


East Coast / West Coast

EAST COAST

DO IT AND DO IT NOW

YOUR PLAN STINKSWHERE IS THE SCHEDULE?

THAT'S TOTALLY INCOMPETENTSHUT UP

WE'RE DONE GET OUT OF MY OFFICE

OH SH-WHAT'S WRONG WITH YOU?

HE'S A SUBORDINATEBOZO

I'LL COVER YOUR REARSHUT UP A MINUTE

DO IT RIGHT OR YOU’RE FIREDPUNCHING HIS/HER LIGHTS OUT

IS THERE A SPEC?WE ARE UNDER BUDGET

WE ARE OVERRUNNING BUDGET WE FINISHED EARLY

UNEMPLOYED

YOU'RE FIRED

GETTING RID OF DEAD WOOD

BUGS

KNOWN TO HAVE NOT IMMEDIATELY FAILED

WEST COAST

CAN YOU GO ALONG WITH THE PROGRAM?

LET ME SHARE MY FEELINGS ON THE PLAN

WHAT IS THE GAME PLAN?

LET ME BUILD ON THAT POINT

THANK YOU FOR SHARING THAT INPUT

HOW DO YOU FEEL ABOUT THAT ?

LET’S GET A CONSENSUS ON THIS ONE

THANKS FOR BRINGING THIS PROBLEM TO MY ATTENTION I CERTAINLY UNDERSTAND WHY YOU FEEL THAT WAY

HE'S A TEAM PLAYERSUBCONTRACTOR

CONSIDER ME YOUR RESOURCE LET ME SHARE THIS WITH YOU

I HAVE CONFIDENCE IN YOUCONSTRUCTIVE CONFRONTATION

WHAT IS A SPEC?WE HAVEN'T STARTED YET

WE ARE ON SCHEDULENO TRANSLATION CONSULTING OR IN TRANSITION

WE NEED MORE DISTANCE IN THIS RELATIONSHIP PLANNED EARLY RETIREMENT

UNINTENDED FEATURESIT WORKS


Industry Differences

� Commercial

� System

� Military

� MIS

� Outsourced

Source: Capers Jones, Software Assessments, Benchmarks, and Best PracticesSource: Gregory Pope, Why Software Practices Become Evil

Many SQA practices are not portableacross industries, i.e. Military to Commercial

9/28/2011

23


Software Development Principles Are Portable

Know who our customers are and meet or exceed their expectations with valuable software systems that are supportable.

Understand the context in which our software systems will operate as well as the technical constraints and interfaces.

Assemble the right team of individuals, give them the environment and support they need, and trust them to get the job done.

Support the customer’s expectations throughout the software project life cycle.

Pursue technical excellence by continuously preventing, detecting, and removing defects from the software system.

Collaborate and coordinate with partners frequently throughout the software project.

Create, integrate, and manage the software codes focusing on continuous improvement of the process and products.

Demonstrate frequently during development that the software system supports the customer’s expectations and is fully supportable.

Source: Gregory Pope, Why Software Practices Become Evil


My Code Does Not Need Testing

But you can run all the experiments you want.

9/28/2011

24


Language Issue:

When Was the First Time You Heard the word “Test”?


A B C

1. Rapid 1. Quality 1. Assurance

2. Unified 2. Verification (and) 2. Validation

3. Agile 3. Experimental 3. Trails

4. Meta 4. Examination 4. Study

5. Flexible 5. Observational 5. Demonstration

6. Tailored 6. Conceptual 6. Prediction

7. Scalable 7. Acceptance 7. Proof

8. Integrated 8. Criterion 8. Scoring

9. Independent 9. Requirement

10. Observed 10. Satisfaction

11. Customer Based

Don’t Call It Testing Table

9/28/2011

25


A B

1. Potential 1. Anomaly

2. Suspect 2. Correctness

3. Tentative 3. Believability

4. Pseudo 4. Certainty

5. Unresolved 5. Convergence

6. Unstable 6. Correlation

7. Irregular 7. Correctitude

8. Arbitrary 8. Correspondence

9. Random 9. Censure

10. Fuzzy 10. Result

11. Biased 11. Presentation

Don’t Call It a Bug Table


“The software was so good that the

developers felt it to be without bugs and not

necessary to test. We did, however, perform

some Rapid Requirement Proofs and found

a number of cases of Irregular Convergence

and Biased Believability. These findings

were handled by the developers as trivial

enhancements, which have now been fully

implemented, and we are ready to ship after

performing the mandatory Independent

Observational Scoring.”

Release Notes:

Source: Gregory Pope, Test Is a Four Letter Word

9/28/2011

26


Solutions

� Harley Davidson

� Cold Stone

� Mongolian Barbeque

� Oakley

� Risk Based Graded Approach

� Crystal

� Let Developers Pick Solutions


Buckshot and Birdshot

9/28/2011

27


Shotgun Analogy

Shot

Size

Pellet

Diameter

(inches)

# of

Pellets

in a 1oz

Shell

000 Buck .36 6

00 Buck .33 8

0 Buck .32 9

1 Buck .30 11

2 Buck .27 15

3 Buck .25 19

4 Buck .24 21

BB .18 50

2 Bird .148 90

4 Bird .129 135

5 Bird .12 170

6 Bird .109 225


Agile Is a Bunch of Things

� Scrum

� Feature List

� Acceleration

� Sprints

� User Stories

� Pairs Programming

� Test Driven Design

� Automated Unit / System Testing

� Continuous Integration

9/28/2011

28


How to Load the Buckshot

� Determine problems, communicate with:

• Developers

• Testers

• Stakeholders, Users

• Marketing

• Help Desk

� Determine problems

� Look up problems to determine solutions

� Load your “shells” with these solutions


How to make Software Better

Neuron Connections

9/28/2011

29


Silver Buckshot Demonstration


Silver Buckshot Demonstration – Problem List

9/28/2011

30


Silver Buckshot Demonstration – Solution List


Problems that Solutions May Solve

9/28/2011

31


Solutions that Prevent Defects

51 52

64

28

1117

22

64

13

28

15

30

16

0

10

20

30

40

50

60

70

Prevention Techniques

Prevention


Solutions that Detect Defects

7

41

9

33

41

19 1813

31

41

13

6 7 6 7

20

0

24

7

41

4

33

17

7

05

1015202530354045

Detection Techniques

Detection

9/28/2011

32


Silver Buckshot Conclusion

� The proverbial Silver Bullet

• No cure alls, but each has contributed

� Most common request

• How to make things better, cheaper, faster

� Definition of Better

• Better for everyone involved

� Good ideas reincarnate

• New names, new groupings

� Some modern challenges

• Security, Complexity, Scale, Safety

� Silver Buckshot – Common problems and solutions

• Buckshot Wiki, expert system on a Wiki

Documents

No Silver Bullet? Silver Buckshot May Work · 2013-08-25 · "No Silver Bullet? Silver Buckshot May Work" Presented by: Gregory Pope Lawrence Livermore National Laboratory Brought