Driving Risks Out of Embedded Automotive Software

Parasoft Copyright © 2016 1

September 30, 2016

Driving Risks out of Embedded Automotive Software


Your Presenters

Arthur Hicken is Chief Evangelist at Parasoft where

he has been involved in automating various software

development and testing practices for over 20 years.

He has worked on projects including cybersecurity,

database development, the software development

lifecycle, web publishing and monitoring, and

integration with legacy systems.

Follow him @codecurmudgeon

Alan Zeichick is Principal Analyst at Camden Associates,

where he focuses on networking technologies, cloud

computing, software development and telecom. In the software

development industry, Alan is well known as the founding

Editor-in-Chief of SD Times and of Software Test &

Performance Magazine. A former mainframe jockey, Alan has

been researching, writing, speaking and consulting on cutting-

edge information technology for more than three decades.

Follow him @zeichick

https://twitter.com/zeichick

https://twitter.com/zeichick


Agenda

Policy enforcement

Reducing defects during coding

Effective techniques for acceptance testing

Using metrics analytics to measure risk

Using SDLC analytics


Open and hide your control panel

Join audio:

• Choose “Mic & Speakers” to use

VoIP

• Choose “Telephone” and dial

using the information provided

Submit questions and comments via

the Questions panel

Note: Today’s presentation is being

recorded and will be provided within

48 hours.

Your Participation

GoToWebinar Housekeeping


Much attention has been paid to hacking


Yet security is only one of the defects

▪ Many cars have more than 100 ECUs▪ 10s of millions of lines of code

▪ Software Everywhere

No visibility into embedded software from vendors

▪ More software than a fighter jet!


It’s a rolling data center

“A typical luxury sedan now contains about 100 megabytes of code that controls 50 to 70 computers inside the car, most of which communicate over a shared internal network.”

- MIT Technology Review


There’s incredible complexity


Complexity breeds errors, bugs, recalls

• Software errors $350/car in 2005 (IEEE)

• Auto problems cost $,$$$,$$$,$$$

• NHTSA estimates $100 per vehicle

• $3 billion annually for recalls/fixes

• Many recent recalls > $1 billion

• Liability above and beyond repairs

• Software is an ever growing percent

• Consequences are serious


We’re not talking about Pokemon Go

▪ Software quality in automotive is life-safety critical

▪ Added complexity introduces new vulnerabilities and safety concerns

▪ Car makers can’t audit or review every line of code

▪ The threats of failure involves lives.

▪ It’s clear: We can’t simply treat QA as in the past — or as primarily a hardware issue


One real-world measure is recalls


Software recalls are a growing problem

▪ With proliferation of ECUs, the problem is exploding From 2005-2012, 32 software recalls affected 3.6

million vehicles From 2012-June 2015, 63 software recalls affected

6.4 millions vehicles From less than 5% of all recalls in 2011, software-

related recalls rose to almost 15% in 2015

In 2011, only 3 software components were recalled; in 2015, it rose to 20 components


Software-related recalls are everywhere


It’s getting worse, and will keep doing so

▪ Defects? Design flaws? Security vulnerability?▪ Does it matter?

August 2016: VW Car-Net shown to be capturing and uploading vehicle sensor data

July 2016: Several incidents with Tesla may be attributable to faulty software

June 2016: Apple patents iPhone Bluetooth method to unlock/start cars remotely

June 2016: Hackers demo remote takeover of Mitsubishi Outlander Plug-In Hybrid


There’s no one source for software flaws


Or even one weak link on the supply chain


And that doesn’t even include connectivity



Poll: Code quality driver

What is the biggest driver of automotive code quality?

Adherence to industry standards like MISRA

Peer review of all source code

Quality contracts with supply chain

A solid process for the SDLC

Context-rich metrics and metadata analysis


Digging deep into policy, practices, metrics

Policies cover the entire SDLC, not just coding and testing

Policies should be defined by architects or group of architects

Architects and managers define the practices required to meet those policies

The organization needs to find tools to automate practices and policy enforcement

Tools like Policy Center can help by syncing what we are doing with what we are measuring with external standards, like ISO 26262 or MISRA

Why? In order to set goals, measure results


Software verification in many stages

Coding standards Unit testing Integration testing Functional testing Memory error detection

Coverage analysis Regression testing Workflow automation Peer code review & document inspections


ISO 26262 Software Tools Map

Number Description Tool Functionality

5.4.6 Correctness of software design and implementation

Static analysis

8.4.4 Design principles for software unit design and implementation

Static analysis

8.4.5 Verification methods Static analysis , flow analysis, peer review

9.4.1-2 Unit test execution Unit testing

9.4.3-4 Unit test creation Unit testing, coverage

9.4.5 Test requirements Requirements management

10.4.2 Integration tests Test execution, coverage

10.4.5 Completeness of integration testing coverage

10.4.7 Requirements for integration test environment Stubs, virtualization/emulation


Where to start

1. Measure software test coverage

2. Improve test coverage

3. Static analysis is an essential tactic

4. Implement preventative coding standards

5. Use runtime memory detection in the test lab

6. Link requirements to code and tests and results

7. Get results data back from supply chain vendors

8. Analyze data collected during development


Poll: When to catch defects

When is the best time to catch a safety-critical software defect?

During the design/architecture phase

While writing the code

When code is checked-in to the repository

During peer review

During testing



Managing all that complexity

Prevention of software flaws Coding Standards

MISRA – software coding standards ISO 16949 - Automotive Quality ISO 26262 – Functional Safety ISO 33001 - Process assessment for software development

Runtime error detection

Integrated development testing results Coding standards, unit testing, coverage,

requirements, static analysis…

Visibility & Traceability


Value of coding standards

The MISRA Guidelines were written specifically for use in systems that

contain a safety aspect to them. The guidelines address potentially unsafe

C language features, and provide programming rules to avoid those

pitfalls.


Static analysis means prevention

Relationship of automated analysis Preventative static analysis

Flow analysis


Uninitialized memory example Runtime will find it IF the test suite is thorough

Flow analysis may find it depending on complexity

Pattern to prevent: Initialize variables upon declaration

Much of MISRA is designed to prevent rather than detect


Standards reduce supply chain risks

Standards and static analysis applied properly prevent errors, reduce risk

Integrated results provides control, measurement, traceability, accountability

Cost of good software is less than the harm caused by a recall or other failure


What’s Needed to Control Risk

A clear sense of ownership and responsibility for quality

Policies that define quality, such as test coverage, conformance with standards, open source licenses

Practices that enforce those policies – like automated static tests or code reviews

Metrics that show compliance – like 90% code coverage in current tests, or 28% failure rate

Definition of “done” – like zero tasks/user stores incomplete, all unit test failures with 5%, etc.

Definition of “done” will change at different times/phases


Validating Requirements

Plan testing for defined Requirements

Test scenario definitions to document use-cases

Confirm Requirement/Test associations in the Requirement Test Matrix report

Automated tests for Requirement Definitions

Code level unit tests

Functional tests

Associate tests with Requirements using annotations

31


Non Functional Requirements for Software


Acceptance Testing

Verify that software development policy is working

Verify that requirements are met

Check for possible failures / flaws

Tests from the outside in

Variety of real-world scenarios

Include security ”penetration” testing


Contractual Software Verification

Coding standards

Unit testing

Memory error detection

Coverage analysis

Regression testing

Peer code review & document inspections


The Problem


What Can You Measure

Code churn

Field bugs

Static analysis findings

Test failures

Coverage

Performance

Counts (lines, files, …)

Bug Arrival Rates


Example: Code Metrics

Do problems indicate simple coder error, or something more?

Prioritize by understanding impacts of policy violations

Understand where complexity means poor design, hard to validate performance or cover with static/dynamic tests

Identify areas with good (or bad) ROI for refactoring

After all, refactoring is not only costly (time and money) but also introduces new risk


Metrics with a Capital M

Some metrics have taken on a life of their

own

Complexity

Cohesion

Coupling

Maintainability

KLOC

There are no silver bullets, no single metric that defines “good” vs. “bad” software


Context comes from metadata

Understand the meaning of the code, modules, tests

Functional vs non-functional requirements

And externalities, such as licenses, budgets, industry standards, as well as audits

For every task there is an assignment (ownership)

As well as budgets, priorities and risk assessments

May be specific security rules.


Checking Your Work

Did you get the right numbers?

Are they going in the right direction?

Are you measuring enough?

Are unexpected things happening?

Are the measurements automatic?

Manual estimates are inconsistent

Multiple layers of manual collection yield

compound rounding errors


Poll: What kind of testing?

Which testing is most valuable in software verification?

Unit testing

Functional testing

Static code analysis

Memory leak detection

Penetration testing


Dashboards Vs Process Intelligence

2001 Ford Explorer

• Isolated data points

• No priority

• Binary warnings (Check

Engine)

• Nothing

• All

2016 Chevy Volt

• Multi-variate analysis

• Customizable

• Engine efficiency

• Valuable data (Range)

• Real-time feedback


Harnessing “Big” Data

Aggregate data

Correlate data

Mine data

Create

Reports

Dashboards

Tasks

Alerts

Continuous testing/delivery/release


Decisions Based on Metrics and Metadata

What’s the priority and cost/benefit of fixing now vs. fixing later vs. letting it slide

Are there bigger problems with code complexity

What about integration with external systems (like TFS)

Architects and top managers configure parameters, to determine how rules are enforced

After all, it’s only partly about functional requirements for the code; non-functional requirements are equally important – or perhaps even more so.

Polices, and thus practices, must be configured to enforce both functional and non-functional requirements


Parasoft Support for Automotive

Policy definition management

Requirement definition and tracking

Static analysis

Unit test

Peer review


Coverage


Conclusions

Safety-critical automotive software issues are going to get worse

• Due to ever-increasing requirements, faster processors, and greater connectivity

This affects software from in-house and the vast supply chain

• The right tests can help make informed decisions faster

Coding standards and practices can drive policies

• And lead to definitions of “done”

Code metrics can help manage risk…

• If they have the right metadata context

• And if integrated across SDLC practices and systems

The result: transparency, feedback, supervision — and managed quality


Blog: http://alm.parasoft.com

Web: http://www.parasoft.com/jsp/resources

Facebook: https://www.facebook.com/parasoftcorporation

Twitter: @Parasoft @CodeCurmudgeon @Zeichick

LinkedIn: http://www.linkedin.com/company/parasoft

Google+ Community: Static Analysis for Fun and Profit

Webinar: RX for FDA Software ComplianceAug 25th

IoT API's session - API World - San Jose, CA Sep 13th

Managing Auto Supply Chain Risk – Automotive Software Kongress - Germany Sep 21-22

http://alm.parasoft.com/

http://www.parasoft.com/jsp/resources

https://www.facebook.com/parasoftcorporation

http://www.linkedin.com/company/parasoft