Design Driven Cost Estimation via Design Driven Testing Doug Rosenberg Copyright © 2011 ICONIX

Design Driven Cost Estimationvia

Design Driven Testing

Doug Rosenberg

Copyright © 2011 ICONIX

“How can we count lines of code before coding?” asked Alice


CoCoMo meets DDT


1987 2011

ICONIX was heavily involved with CoCoMo from the mid-

80s to the mid-90s

• We sold a CoCoMo tool for about 10 years• What did we learn?• A) Estimation is a lot easier when you

know how many lines of code you will be building

• B) It isn’t always easy to know how many LOC you’ll have up front

• Our CoCoPro tool calibrated estimations to a history file of past projects, to converge to better estimation results over time


We know how to count code after it’s written


• Q: how can we get a line of code count before coding? • A: Count lines of unit test code, generated from model

Cost estimation via testing?Isn’t that backwards???

• In today’s “agile” universe, tests are sometimes written before code (Test Driven Development)

• With Design Driven Testing we generate unit test code from UML/SysML models before code is written

• We can line-count this unit test code and use it as a basis for estimation



DDT flips “agile” test driven development around

• That’s good because driving design from unit tests is 180 degrees backwards

• It’s better to drive testing from design

• Cost estimation can be driven in the same way


DDT addresses unit testing by developers and acceptance

testing by QA folks

• We’ll focus on unit testing in this talk

• DDT automates generation of skeleton unit test code from UML models

• This unit test code can be line-counted and used as a basis for LOC estimates


Unit tests are auto-generated from sequence diagrams

• This is good because we can count lines of unit test code and use it as a basis for estimation, before code is written

• How does it work?


1. Generate a test case for each message on a sequence diagram


2. Define test scenarios for each test case

Test with good data, test with bad data, etc.

Define as many scenarios as are needed to fully exercise the operation being tested

More complex operations will have more test scenarios


3. Transform test cases to unit test classes

• A test method is created for each test scenario

• These test methods can be code generated


Transforms are instantaneous

• setUp and tearDown methods are also generated, in a addition to a method per test scenario

• Automation is key to the whole DDT approach because it’s too much work if it’s not automated


4. Test class -> test code

• Unit test code is generated automatically

• The amount of code is proportional to the complexity of the operation being tested

• This unit test code can be line-counted and used as a basis for COCOMO estimation

• NOTE: Not a 1:1 ratio between unit test code and production code


We have an early-lifecycle line of code count (before coding begins)

This line of code count is directly generated from the UML or SysML model, and is directly proportional to the complexity of the code that will be written, because more complex methods will have more test scenarios

No handwaving


Oh, BTW…No gaps in test coverage

• Minor side benefits of DDT include better designs and better test coverage


What if we could start from an earlier lifecycle artifact than

sequence diagrams?

• Sequence diagrams represent a very detailed level of design.

• We probably need to estimate earlier in the lifecycle.

• We can generate test code from conceptual designs, as soon as use cases are defined


Controllers are testable units of system behavior

• We identify logical software functions on robustness diagrams as a disambiguation device when writing use cases

• Each controller may expand to several messages on a sequence diagram

• Fewer tests are required• Controller tests can be

code generated just like sequence message tests


Breadth-first test coverage across a scenario

• And an earlier-lifecycle artifact to drive test code from

• This can provide a “first guess” estimate

• Re-estimation during detailed design will provide more accurate estimates


Add test scenarios for each test case as before

• The conceptual design will not include the “infrastructure” or “scaffolding” code, but will reflect the core functionality that needs to be built

• This is OK for a first-pass estimate


Transform to classes and generate code

Since robustness diagrams are drawn just after use cases are written, we have a line-countable code base very early in the project lifecycle

The earlier we can estimate accurately, the better we can plan budgets and schedules

What about hardware/software

systems?

• Does all this still work if we’re modeling an embedded system using SysML?

• Yep.


DDT/Systems extends DDT/Software for SysML• DDT/Software is a great place to

start system testing, because most systems are software intensive

• DDT/Software (DDT) includes both Developer Testing and QA/Acceptance Testing

• DDT Scenario Testing applies directly to “system of systems” testing


What is DDT/Systems?

• A comprehensive approach to validation and verification for systems and system-of-systems

• A systematic way to drive V&V from SysML models

• An extension of Design Driven Testing for software to include full HW/SW system V&V


DDT/Systems extends DDT for the full range of SysML models,

including interface and state testing


SysML Example Model: Pothole detecting spacecraft

• Orbits the earth• Takes pictures of city streets• Downlinks images to ground station• Detects potholes via image analysis• Emails images with annotated

potholes to subscribers• (Thanks to Dr. Mike Sievers of

NASA/JPL for his help with the

POTHOLE example, and with

defining DDT/Systems)


How do we V&V our spacecraft from this SysML model?

• Generate tests from requirements• Generate interface tests from ports• Generate scenario tests from use cases• Generate state machine tests from triggers,

do/entry/exit behaviors• Generate unit tests from sequence messages• Generate unit tests from class operations• Generate parametric tests


We’ll have one sequence diagram per use case


Sequence diagrams show all system behavior within each use case


These test cases are transformed into unit test code and can be an input to CoCoMo cost models


State machines include triggers, and entry/exit/do behaviors

Thanks to Dr. Mike Sievers of NASA/JPL for assistance


We can generate all of these behavior tests automatically from a SysML model

Thanks to Dr. Mike Sievers of NASA/JPL for assistanceCopyright © 2011 ICONIX

Interface tests are generated from Ports



Interface Tests for Non Volatile Memory include testing the Buffer Control Interface, Instrument Data Input, and Xband Output



Current Status

• DDT is described in Design Driven Testing (Apress 2010)

• DDT/Systems is described in ICONIX Process Roadmaps (Fingerpress 2011)

• DDT/Software currently automated in Sparx Enterprise Architect; DDT/Systems extensions not yet released

• MagicDraw plug-in currently under development, scheduled release 4Q 2011



Summing up

Early lifecycle line of code estimates are generally beneficial for cost estimation

Design Driven Testing provides a basis for early LOC estimates from conceptual and detailed design views of a use case

DDT/Systems extends this to cover state and interface testing from SysML models

LOC estimates are generated directly from the design models, thus all handwaving is eliminated

WAGs become substantially more scientific (SWAGs are better than WAGs)


Questions?

[email protected]

[email protected]

[email protected]

mailto:[email protected]




Thank You

Documents

Design Driven Cost Estimation via Design Driven Testing Doug Rosenberg Copyright © 2011 ICONIX