Static analysis works for mission-critical systems, why not yours?

1© 2017 Rogue Wave Software, Inc. All Rights Reserved.

1

Confronting the mission-critical software testing challengeEpisode 2:

Static analysis works for mission-critical systems, why not yours?

Walter CapitaniProduct manager, Klocwork


2

Presenter

Walter CapitaniProduct manager, KlocworkRogue Wave [email protected]: @walter_capitani

mailto:[email protected]


3

1. How do you select a static code analysis tool?

2. What kind of defects/issues are you looking for?

3. How/when/where should you deploy static code analysis?

4. Common myths and barriers to adoption

5. Q&A

Agenda


4

Poll #1What is the primary method you use to test code?• Code reviews• Unit tests• Manual tests at build time• Automated tests at build time• Automated testing using CI tools


5

How do you select a static code analysis tool?


6

Decision metrics for static code analysis

• What kind of defects are you looking for?

Security issues

Memory leaks

Applicationcrashes

Otherdefects

Improve quality

Enforcecompliance

Improvesecurity Other

• What are you trying to accomplish?


7

What kind of defects/issues are you looking for?


8

• Find common issues in code

• Not easy to spot with the human eye– Not generally found by code review– Many are traditionally found with dynamic testing after a

failure has occurred in testing or the field

What kind of defects are we looking for?

Buffer overflows

Security exploit or program crashes

Null pointer dereferences

Your program crashes

Memory leaks

Processor runs out of memory and locks up

Uninitialized data usage

Data injection

Platform/OS specifics

Privilege escalation, etc.

Concurrency

Deadlock

Suspicious coding

practicesVariable assignments, function calls


9

What is static code analysis?

Performs one or more processes

Syntax Analysis

Data Flow Analysis

Symbolic Logic Analysis

Requires source code

The most accurate tools must be able to compile the code

No changes to your existing build flow

Different types of analysis

Intra-procedural (simplest analysis)

Inter-procedural

Inter-file


10

Syntax Analysis

• Creates a lossless transformation of the source code• Generates the ‘Abstract Syntax Tree’• Can be used to find Coding Style Issues and Simple Defects

– Simple security defects (e.g. use of banned encryption API)– Simple coding style issues (e.g. no dynamic memory

allocation)This function

allocates memory

Name = “malloc”,

Source Code Abstract Syntax Tree


11

Syntax Analysis -example

if(i = j) j++;

if(i == j) j++;

Defect: Assignment operator used in

conditional statement

Assignment operator replaced with intended comparison operator

Vulnerable Code

Fixed Code


12

This seems to work well, but…

• These defects are contained in a single program statement

• They are not dependent on values from external functions

• Syntax Analysis can only find a limited set of defects

To find more interesting defects you need to perform

more sophisticated analysis


13

Data Flow Analysis

• Monitoring of the lifecycleof data objects:– Creation– Assignment– Usage– Deletion

• Must be monitored across allpaths in the Control Flow Graph– Function calls– Compilation units

• Can find program crashes across functions and files


14

Data Flow Analysis - example

• This function a()will cause the program to crash at line 3

• This function g() will cause the program to crash if position is outside the valid range – how do we know if this will happen?

1 void a(){2 int buffer[32]; // valid range of 0..312 buffer[35] = 5; // buffer access outside valid range (35)4 return;5 }

1 void g(int position, int value){2 int buffer[32]; // valid range of 0..313 buffer[position] = value;4 return;5 }

3 buffer[35] = 5; // buffer access outside valid range (35)

Defect: Array bounds violation


15


• Data Flow Analysis tracks what potential values are actually used when function f() calls function g()


1 void f(){2 g(10,55); // calls function g with position=10, value=55 3 return;4 }

No defect: values within valid range

Vulnerable Code


16

1 void h(){2 g(35,25); // calls function g with position=35, value=25 3 return;3 }


• Data Flow Analysis tracks what potential values are actually used when function h() calls function g()


3 buffer[position] = value; // buffer access outside valid range (35)

Defect: Array bounds violation (program

crash)

Vulnerable Code

2 g(35,25); // calls function g with position=35, value=25


17

1 void h(){2 g(35,25); // calls function g with position=35, value=25 3 return;3 }


• Data Flow Analysis tracks what potential values are actually used when function h() calls function g()

1 void g(int position, int value){2 int buffer[32]; // valid range of 0..313 if (position < 0 || position >31 0) // Check position is valid4 return;5 buffer[position] = value;6 return;7 }

No defect

Fixed Code

2 g(35,25); // calls function g with position=35, value=25

3 if (position < 0 || position > 31) // Check position is valid4 return;


18

This also seems to work well, but…

• Data Flow Analysis alone can only understand actual numeric values (or ranges of values)

• What if there are no numeric values at all? How do we determine valid data flow paths?

To find more interesting defects you need to augment data flow analysis

with Symbolic Logic


19

Symbolic Logic

• Define functional behavior between symbols• Don’t necessarily know what the values will be at runtime• Used to infer software behavior

1 void f(int i, int j){2 int buffer[32]; // valid range of 0..313 i = j;4 5 /* set the value of k */6 if (i == j)7 k = get_tainted_data(); // Since i equals j, k is tainted8 else9 k = 0;10 11 /* read the value of k */12 if (i != j) // Since i = j, k will not be used13 buffer[k] = 0;14 return;15 }

3 i = j;

7 k = get_tainted_data(); // Since i equals j, k is tainted

12 if (i != j) // Since i == j, k will not be used


20

Symbolic Logic

• Symbolic logic determines that since i = j, there is no use of tainted data at line 13

• Otherwise a tool must “guess” at the defect


3 i = j;

7 k = get_tainted_data(); // Since i equals j, k is tainted

12 if (i != j) // Since i == j, k will not be used


21

Symbolic Logic

• If we change line 12, then a defect appears!


12 if (i == j) // Since i == j, k will be used

Defect: Unvalidated input in array index (program

crash)

Vulnerable Code


22

How/when/where should you deploy static code analysis?


23

What are we trying to accomplish?

• This will guide what kind of implementations our static code analysis tools should support

Improve quality

Enforcecompliance

Improvesecurity


24

Frequency of analysis

Once per release

Every check-in Continuous integration

Nightly/Weekly

Developer desktop

Can be used to ensure that no issues are introduced with any check-in

Good for security compliance, minimizes backlog of work to do in release phase

Detect issues as they are typed Most efficient method to save developers’ time

Typically used for compliance purposes, limited value to improving code


25

Development Cycle

Edit Save Compile Test Check In Build Analyze

& Fix

• Late stage “rework” reduces tool adoption• Timelines compromised• Issues are more expensive to fix

Traditional analysis done after compile/build


26

Eliminates new defects from being checked back into the team level build

No extra work for developers In-context checking and fixes Continuity of development flow

Edit Save Analyze& Fix Compile Test Check

In Build

Development Cycle

Best practice: Analysis earlier in the cycle


27

Improves coding practices Alerts the developer immediately when they enter a defect Provides entire path from “source to sink” of how the issue occurs Provide help on how to remedy Provides links to the specific coding standards that may be

violated Allows you to edit and customize that advice with simple HTML

editing.

The key is that not only do we help the developer, by telling them immediately it is an excellent “teachable moment.”

Finally, since the developer makes the fix immediately, your code base is never impacted.

Edit Save Analyze& Fix Compile Test Check

In Build

Development CycleDesktop analysis advantages


28

Common myths and barriers to adoption


29

Poll #2What are the barriers to adoption in your organization?

• Cost of the SCA tool• Lack of perceived value to developers• To complex to integrate• Too many false positives• No barrier – we use an SCA tool already


30

#1: The truth about false positives

• All automated safety systems have false positives– That what static code analysis is:

an automated safety system for your software developers

• Safety systems in automobiles:– Blind Spot Detection systems– Back up sensors

• All of these systems will generate erroneous warnings sometimes

• As long as the Signal-to-Noise ratio is reasonable, and we can tune the system to generate the results we are looking for, these systems add value


31

Common sources of false positives

• Enabling the wrong defect checkers• “Developer false positives”• Third party libraries


32

How to manage false positives

• Refer to our decision criteria– Look at defects that serve your purpose:

• Improve Security Enable security defects• Improve Quality Enable memory leak defects

• Prioritize what defects specific developers should look at:– Advanced static code analysis tools provide different ways

for developers to view defects

• Tune the static code analysis tool to reduce or eliminate defects from third party libraries


33

#2: Static code analysis is for junior developers

• If that were true, it would mean software written by senior developers would be bug-free

• Sophisticated tools with data flow analysis find issues that may even get past senior developers, particularly in large complex code bases (what about 3rd party and legacy code)

• Static code analysis tools evolve over time to find new security defects – even senior developers will need training to spot these new threats…


34

#3: Should test and QA find bugs?

• 80% of defects are introduced in development

• Each defect found in test costs 50x to fix

• Test and QA can spend more time trying to make the product better, rather than reporting issues that could have been found at the development stage


35

Summary


36

Summary

• Select a tool that supports the corresponding depth of analysis

Decide what kind of defects you need to find

• Select a tool that supports the appropriate workflow

Decide what you are trying to accomplish

Proper configuration and tuning of the SCA tool helps with developer adoption

1

2

3


37

Q&A


38

Follow up

Free e-book:

Building better code with static code analysis

www.roguewave.com/programs/building-better-code-with-sca

Learn more about Klocwork static code analysis:

Kate AndreevaInside Sales Account Executive

[email protected]

http://www.roguewave.com/programs/building-better-code-with-sca

http://www.roguewave.com/programs/building-better-code-with-sca

mailto:[email protected]


39

Stay tunedConfronting the mission-critical software testing

challengeFeb. 22: What if you could eliminate the hidden costs of development?Combat different types of development inefficiency by examining error-prone tasks, waiting for resources, “bug fix crowdsourcing,” and more to learn what the industry is doing about them and what you can do to get ahead.

Available on-demand www.roguewave.com/scaEpisode 1: How to achieve security, reliability, and productivity in less timeEpisode 2: Static analysis works for mission-critical systems, why not yours? (Soon!)

http://www.roguewave.com/sca


40