Test-Oriented Requirements Engineering · Test-Oriented Requirements Engineering REConf, Munich 18. March 2015 Christof Ebert and Harry Sneed

Test-Oriented Requirements Engineering

REConf, Munich 18. March 2015Christof Ebert and Harry Sneed

© 2015 . Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V 1.0. 2015-03-18

2/37

Vector Consulting Services

… supports clients worldwide in improving their product development and IT and with interim management

… with clients such as Accenture, Audi, BMW, Bosch, Daimler, Hyundai, IBM, Lufthansa, Munich RE, Porsche, Siemens, Thales, Vodafone, and ZF

… offers with the Vector Group a portfolio of tools, software components and services for engineering

… is globally present as a group with over 1300 employees and well over 250 Mio €

www.vector.com/consulting www.vector.com/PREEvision

Railway &Transportation

IT

Automotive

Aviation & Defense

Energy &Environment

Medical &Healthcare


3/37

Prof. Dr. Christof EBERT

Christof Ebert is managing director at Vector Consulting Services.

He supports clients around the world to improve product strategy and product development and to manage organizational changes. Prior to that, he held global management positions for ten years at Alcatel, at that time an IT world leader.

A trusted advisor for companies around the world and a member of several of industry boards, he is a professor at the University of Stuttgart. He authored several books including the leading industry reference on requirements engineering published by dPunkt.

[email protected] www.vector.com/consulting


4/37

Harry SNEED

Harry Sneed has been in the IT field since 1967, when he started as a Fortran programmer for the U.S. Navy. During this time he has gathered a wealth of experience as a programmer, analyst, project manager, laboratory manager, consultant, manager, trainer, university lecturer, researcher and author.

His focus of interest is on software management, metrics, software and testing. Besides his project work, he has written over 400 articles and 22 books on different aspects of software engineering. He is currently consulting a large migration project in Austria while teaching at the universities of Regensburg, Dresden, Vienna and Hagenberg.

[email protected]


5/37

Structure of Presentation

Motivation


Case Study

Conclusions


6/37

Business Challenges – Vector Client Survey 2015

Innovative products

Others

Efficiency

Distributed development

Cost reductionBig data

Governance

ComplexityManagement

Robust products

Connectivity (e.g. PLM/ALM, infrastructure)

0%

10%

20%

30%

40%

50%

60%

0% 10% 20% 30% 40% 50% 60%

Important forown responsibility

Important forown industry

Vector Client Survey 2015 ; www.vector.com/trends ; Sum > 100% because 3 answers per question were allowed

Evolution since 2014: Complexity management and connectivity play a dominant role, while before the major focus was on cost reduction.


7/37

It all Starts with Insufficient Requirements

Value orientation: Ca. 50% of all functions in a product are not used; only 20% of all functions actually deliver value

Early defect removal: 80% of all defects detected in test and 43% of all defects in field result from insufficient requirements.

Cost reduction: Typically 3-6% of effort goes into requirements engineering. 40-60% of effort is spent on testing. Doubling RE effort reduces life-cycle cost by 20-40%.

Sources: Ebert 2014, Standish Group 2009, Univ. Karlsruhe 2005, IAG Business Analysis Benchmark 2008 (103 companies, avg. 3 Mio US$ project volume)

Project time

Target

Good RE

Insufficient RE

20-40% efficiencypotential from RE

Projectcost


8/37

Example: Requirements need Testing

Requirement: "When servicing the elevator, configuration data

is automatically presented in the same way as last time"

Business need: Improve workflow management. An optimal setting

when working on configuration data isavailable from the most recent use.

This requirement is very vague. Here some review findings:

Settings per user or freeze settings per elevator? Can views of the documentation be defined freely? Are all views available to all users?

Derived test case: Change configuration display (zoom, window arrangement, clippings, etc.).

Exit the elevator configuration description. Enter again the configuration description. The description is displayed exactly the same.


9/37

Motivation


Case Study

Conclusions


10/37

Foundations: Good Requirements

A good requirement describes something (function, characteristic, attribute), that is desired, verifiable and feasible.

Need: what will happen if the requirement is not taken into account? What other requirements determine this requirement? What marginal utility does this requirement deliver? There is often no answer and the requirement obviously is not so relevant.

Verifiable: when the requirement is documented it must be clear how it will be tested and accepted at a later stage.

Feasible: requirements must be technically feasible. They do not conflict with other requirements or constraints such as budget or handover date.

A good requirement is test-oriented


11/37

Foundations: Test-Driven Development

Essence: Whenever a requirement is developed the criteria for the test of that requirement must also be stated

Manage change: Whenever a requirement changes it effects those code blocks traversed by its test cases.

Efficient development: Regression test based on automated test. Traceability shows impacts of changes – in both directions.

Req. Test CodeSpecs Cases

Objective: Begin with test in mind


12/37

Conclusion: Test-Oriented Requirements Engineering

Strategy Concept EvolutionIntegration, certification

Development

Test-oriented RE

Typical scope of testing

Intelligent testing implies a strategic focus across the life-cycle connecting concept, engineering and evolution


15/37

Practice: Test-Oriented Requirements Engineering

TC1 passedTC2 openTC3 failed...

Tester Customer

Cross-fertilizationTraceability

RequirementsAnalysis

ComponentTest

SystemTest

SystemDesign

Component Design

Component Implementation

SystemIntegration

Requirementsanalysis

Componenttest

System test,Acceptance

System design

Componentdesign

Componentimplementation

Systemintegration


16/37

Example: Requirement with Acceptance Criteria

Acceptance Criteria:[RF-10] Feed Data Input

Feed data has different correctness. This correctness must be highlighted to the user by the following

color code in the “create event mask” (see Figure 2):

Brown: Only one feed source is available for the event – a comparison with other feed providers is not possible.

Green: all feed sources have the same data

Red: at least one feed source provides other data

Blue: at least one feed source provides other data, but the user has chosen a primary source (see also [RF 22])

Gray: The event is already generated.

This color also takes place if the event is created manually and first feed come afterwards.

This color works only for the main booking.

Acceptance Criteria:[AC-10] Test Feed Data Input

Imagine a league which is covered by Provider A and B.

If Provider A and Provider B have the same start time (e.g. 18:30). Then line is marked green.

If Provider A and Provider B have different start times (e.g. 18:30 vs. 19:00) the line is marked red.

If the start times differ, the user can choose one primary source (if not already defined for the league

in admin frontend) e.g. 18:30. Then the line is marked blue and the event stored with this start time.


17/37

Test CasesTestCase 1Test Operation A

Generate Request AValidate Response A

Test Operation CGenerate Request CValidate Response C

Test Operation BGenerate Request BValidate Response B

TestCase 2Test Operation A

Generate Request AValidate Response A

Business GoalsFunctional RequirementsNon-Functional RequirementsData RequirementsBusiness ObjectsBusiness RulesService Interface DefinitionOther InterfacesService ActorsUse Cases

References to Business GoalsReferences to RequirementsReferences to Business RulesReferences to Business ObjectsReferences to Service InterfacesReferences to other InterfacesReference to ActorsTriggerPre-ConditionsPost-ConditionsMain Path with StepsAlternate Path with StepsExceptions

Use Cases fulfill Requirementsimplement Business

Rules process Objectsinvoke Operationssend Requestsreceive Responses

Requirement Document

Example: Contents of a Service Requirement Specification


18/37

Data

Interface

Definition

InterfaceDefinition

with sample

Data

System Interface Definitionis copied over into theRequirement Document

Tester fills sample Test data into the Table

Data Structure +Test data +Test conditions =Test cases

Goals

Requirements

Business Rules

Business Objects

Use Cases

Example: Contents of a Service Requirement Specification


19/37

Seven Tips for Test-oriented Requirements

1. Each single functional requirement is accompanied by an acceptance criteria which is either fulfilled or not fulfilled.

2. Each single quality requirement is stated with numeric values which can be measured.

3. Business rules are defined so they can be determined to be true or false.

4. Business and data objects are defined with all their attributes, types and states so these can be set and validated at test time.

5. System interfaces, e.g. GUIs, reports and service interfaces, are included in the requirement document so that they can be assigned values.

6. All use cases have pre- and post-conditions which can generated and validated.

7. All text is marked up so that it can be processed automatically to generate test cases.

Build test into the requirement document from the very beginning


21/37

The Challenge: Insufficient Test Competence and Capacity

Problem: Testers in the hot phase of the previous project are also required in the

requirements phase of the next project.

Mitigation: Dedicated teams of expert testers are selected and a certain percentage

included in the plans for each proposed project. Tester optimizes test strategy (i.e., eliminate redundancies, unbalanced

criticality-based testing)

Requirements TestProject 1

Project 2Requirements Test

Bottleneck


22/37

Motivation


Case Study

Conclusions


23/37

Case Study: Test-Oriented Requirements Engineering

RequirementsBusiness Processes

Testfälle, Bug-Tracking

Tool support

Workflow

Consistency with single source

Quality checks


24/37

Articles Custo-mers Suppliers Prices

GetTypes

QueryArticles

ProcessOrder

Back Order

ProcessResupply Billing Dispatch

Back Orders

ResupplItems

BillingItems

DispatchOrders

Back orders

SupplyOrders Invoice Dispatch

Report

Front-End Web Clients

BackEnd Web Services

Persistent Data Tables

Transient Interface Files

System Reports

TriggerReport

BackEnd Web Client

Case Study: System Overview


25/37

Case Study: Functional Requirements

FUNC-REQ-03 Dispatching

For every article item fulfilled a dispatch order is generated and sent to the dispatch office.

The ordered item is sent to the customer within one day after it has been ordered.

Acceptance Criteria:

If article amount > amount ordered, there is a valid dispatch order for this article.

FUNC-REQ-04 Billing

For every customer order in which at least one item is fulfilled, an invoice with items ordered, their prices, the total sum, the VAT and the total amount due is printed and sent to the customer.


For every order fulfilled there is an invoice a billing item for each order item.

The total amount due = sum of all billing item amounts * VAT.

FUNC-REQ-05 Resupplying

The system monitors the articles on stock. If the quantity of an article on stock falls below the minimum quantity that article is automatically reordered. The generated supply orders are collected.

Once a week, at the end of the week, they are processed.

When a supply order is sent out the cheapest supplier at that moment in time is selected.


If article amount < minimum quantity there is a supply order to the cheapest supplier.


26/37

Case Study: Quality Requirements

NF-REQ-01 ResponseTime

The response time for a customer query is <= 1 second.

The response time for a customer order is <= 3 seconds.

Test most complex order with 10 items and measure time from entry to return.

NF-REQ-02 TransactionCapacity

The system is able to process at least 1000 orders an hour without any loss of performance.

Test with 1000 orders within one hour.

NF-REQ-03 Availability

The system is available 24 hours a day, 7 days a week, for at least 95% of the time.

NF-REQ-04 Security

Only authorized customers have access to the system.

Unauthorized persons are blocked from access.

Of 10 attempts to break into the system by unauthorized persons, at least 9 are thwarted.

Test with 10 unauthorized users.

NF-REQ-08 Maintainability

The system source code must have a maintainability index of at least “0.5” from the code audit.

Measure maintainability of code with CodeAuditor.

NF-REQ-09 Reusability

The system source code must have a reusability rate of at least “0.6” from the code audit.


27/37

Case Study: Sample Use Case for Billing

Attributes DescriptionLabel: BillingFulfill: Func-Req-04.Process: Bo-01.Input: GUI-Billing-Items.Output: Invoice, Protocol.Implement: Br-09, Br-10.Trigger: EndofMonth.Actor: AccountantFrequency: MonthlyPreConditions: At least one billing item must be available.PostConditions: An invoice is printed for each customer who ordered in the past month.

MainPath:

1) Billing job is started.

2) System sorts the billing items by customer.

3) System gets address data for each customer.

4) System computes the item cost for each billing item.

5) System sums up the total cost from the item costs of each billing item.

6) System computes the VAT of the total cost.

7) System calculates the data due.8) System prints an invoice for each customer.

AlternatePath: None

Exceptions: System finds no billing items.System finds no customer data for an order.

Rules: Due date = current date + 30 days.VAT = total cost + (total cost * 0.20).

Objects: Billing items, Customer, Invoice.


28/37

Case Study: Logical Test Cases extracted from Requirements

+-----------+--------+--------------------------------+---------+----------------------------------------------------

TestCase-Id BaseType Requirement/Use Case Title TestType TestCase Description

+-----------+--------+--------------------------------+---------+----------------------------------------------------

ORDERS0011 Require FUNC-REQ-03 Dispatching rule For every <article_item> fulfilled a <dispatch_order>

is generated and sent to the <dispatch_office>.

ORDERS0012 Require FUNC-REQ-03 Dispatching rule The ordered <item> is sent to the <customer>

within one <day> after it has been ordered.

ORDERS0013 Require FUNC-REQ-03 Dispatching action It is sent together with the <order_confirmation>.

+-----------+--------+--------------------------------+---------+-----------------------------------------------------

TestCase-Id BaseType Requirement/Use Case Title TestType TestCase Description

+-----------+--------+--------------------------------+---------+-----------------------------------------------------

ORDERS0014 Require FUNC-REQ-04 Billing state The <invoice> includes the <items> ordered,

their proces, the <total_sum>, the <vat> and the <amount_due>.

ORDERS0015 Require FUNC-REQ-04 Billing action All <invoices> is checked by the <accountant>

before being sent out.

ORDERS0016 Require FUNC-REQ-04 Billing action For every <customer_order> in which at least one

<item> is fulfilled, an <invoice> is printed and sent to the <customer>.

ORDERS0017 Require FUNC-REQ-04 Billing action The <invoices> are prepared once a <month>

at the end of the <month>.

+-----------+--------+----------------------------------------+---------+--------------------------------------------


29/37

Case Study: Generated Logical Test Case Table


30/37

Case Study: From Test Cases to Test Scripts

// First Request to Frontend to order Articles if ( operation = "GetTypes");

if ( request = "GetTypes1Request");assert inp.GetTypes1Request_DummyParam = "?";

endRequest ;if ( response = "GetTypes1Response");

assert out.$ResponseTime < "1000"; if ( object = "return" occurs = "2");

assert out.item[1] = old.item[1];endObject;

endResponse ;endOperation;if ( operation = "QueryAvailableProducts");

if ( request = "QueryAvailableProducts2Request");assert inp.CustNo = "009999";assert inp.ArtType = "BOOK";

endRequest ;if ( response = "QueryAvailableProducts2Response");

assert out.$ResponseTime < "1000"; if ( object = "return" occurs = "1");

if ( object = "item[1]");assert out.ResponseArtNo = "4711";assert out.ResponseArtType = "BOOK";


31/37

+----------------------------------------------------------------------------------------------+| TESTPROC: ORDERTEST-TEST DATE: 29.12.13 || SYSTEM: ORDERS PAGE: 1 |+----------------------------------+----------------------------------+------------------------+| Requirement Name | in Document Section | TestCase Name |+----------------------------------+----------------------------------+------------------------+| FUNC-REQ-01 ArticleDisplay | OrderEntry | ORDERS0001 || FUNC-REQ-01 ArticleDisplay | OrderEntry | ORDERS0002 |+----------------------------------+----------------------------------+------------------------+| FUNC-REQ-02 OrderProcessing | OrderEntry | ORDERS0003 || FUNC-REQ-02 OrderProcessing | OrderEntry | ORDERS0004 || FUNC-REQ-02 OrderProcessing | OrderEntry | ORDERS0005 || FUNC-REQ-02 OrderProcessing | OrderEntry | ORDERS0006 || FUNC-REQ-02 OrderProcessing | OrderEntry | ORDERS0007 || FUNC-REQ-02 OrderProcessing | OrderEntry | ORDERS0008 || FUNC-REQ-02 OrderProcessing | OrderEntry | ORDERS0009 |+----------------------------------+----------------------------------+------------------------+| FUNC-REQ-03 Dispatching | OrderEntry | ORDERS0010 || FUNC-REQ-03 Dispatching | OrderEntry | ORDERS0011 |+----------------------------------+----------------------------------+------------------------+| FUNC-REQ-04 Billing | OrderEntry | ORDERS0012 || FUNC-REQ-04 Billing | OrderEntry | ORDERS0013 || FUNC-REQ-04 Billing | OrderEntry | ORDERS0014 || FUNC-REQ-04 Billing | OrderEntry | ORDERS0015 |+----------------------------------+----------------------------------+------------------------+| FUNC-REQ-05 Resupplying | OrderEntry | ORDERS0016 || FUNC-REQ-05 Resupplying | OrderEntry | ORDERS0017 || FUNC-REQ-05 Resupplying | OrderEntry | ORDERS0018 |+----------------------------------+----------------------------------+------------------------+| FUNC-REQ-06 BackOrderProcessing | OrderEntry | ORDERS0021 || FUNC-REQ-06 BackOrderProcessing | OrderEntry | ORDERS0022 |+----------------------------------+----------------------------------+------------------------+| NF-REQ-01 ReponseTime | OrderEntry | Testcase is missing! |+----------------------------------+----------------------------------+------------------------+| NF-REQ-02 TransactionCapacity | OrderEntry | ORDERS0027 |+----------------------------------+----------------------------------+------------------------+

Case Study: Traceability from Requirements to Test Cases


32/37

+----------------------------------------------------------------------------------------------+| TESTPROC: ORDERTEST-TEST DATE: 29.12.13 || SYSTEM: ORDERS PAGE: 1 |+----------------------------------+----------------------------------+------------------------+| Use Case Name | in Document Section | TestCase Name |+----------------------------------+----------------------------------+------------------------+| Customer_Query | OrderEntry | ORDERS0065 || Customer_Query | OrderEntry | ORDERS0066 || Customer_Query | OrderEntry | ORDERS0067 || Customer_Query | OrderEntry | ORDERS0068 || Customer_Query | OrderEntry | ORDERS0069 || Customer_Query | OrderEntry | ORDERS0070 |+----------------------------------+----------------------------------+------------------------+| Customer_Order_Processing | OrderEntry | ORDERS0071 || Customer_Order_Processing | OrderEntry | ORDERS0072 || Customer_Order_Processing | OrderEntry | ORDERS0073 || Customer_Order_Processing | OrderEntry | ORDERS0074 || Customer_Order_Processing | OrderEntry | ORDERS0075 || Customer_Order_Processing | OrderEntry | ORDERS0076 || Customer_Order_Processing | OrderEntry | ORDERS0077 || Customer_Order_Processing | OrderEntry | ORDERS0078 || Customer_Order_Processing | OrderEntry | ORDERS0079 || Customer_Order_Processing | OrderEntry | ORDERS0080 || Customer_Order_Processing | OrderEntry | ORDERS0081 || Customer_Order_Processing | OrderEntry | ORDERS0082 || Customer_Order_Processing | OrderEntry | ORDERS0083 || Customer_Order_Processing | OrderEntry | ORDERS0084 || Customer_Order_Processing | OrderEntry | ORDERS0085 || Customer_Order_Processing | OrderEntry | ORDERS0086 || Customer_Order_Processing | OrderEntry | ORDERS0087 || Customer_Order_Processing | OrderEntry | ORDERS0088 |+----------------------------------+----------------------------------+------------------------+| Dispatching | OrderEntry | ORDERS0113 || Dispatching | OrderEntry | ORDERS0114 || Dispatching | OrderEntry | ORDERS0115 || Dispatching | OrderEntry | ORDERS0116 |

Case Study: Traceability from Use Cases to Test Cases


33/37

Motivation


Case Study

Conclusions


35/37

Test-oriented Requirements Engineering

Early test case specification is part of requirements engineering.

Better requirements quality and balanced test effort due to front loading

Less project failures and quality problems in the field by driving a risk-oriented requirements engineering

Effort reduction with requirements changes by immediate and consistent traceability

Testing effort is reduced by ca. 30% over the life-cycle

Overall cycle time for development projects is decreasing


36/37

Further Reading

Requirements Engineering

Christof Ebertdpunkt.verlag450 pages5. fully reworked edition 2014Chinese edition, 2013

„The classic for systematically handling requirements. Written by a practitioner for practice – easy to understand and to apply! During a joint project I experienced first hand that the author knows what he talks about."Hans Leibbrand, COO, Thales

Contact us – We would be happy to support you!

Phone +49 711 80670-0 www.vector.com/consultingFax +49 711 80670-444 [email protected]

Documents

Test-Oriented Requirements Engineering · Test-Oriented Requirements Engineering REConf, Munich 18. March 2015 Christof Ebert and Harry Sneed