Backward ThinkingBackward ThinkingDesign QA System to Meet Quality GoalsDesign QA System to Meet Quality Goals

Liang Gao

Types of Testing Types of Testing • User testing• Alpha testing• Beta testing• Function testing• Integration testing• Domain testing• Boundary testing• Best representative testing• Logic testing• State-based testing• Path testing• Performance testing

2

• Specification-based testing• Requirements-based

testing• Combination testing• Regression testing• Scripted testing• Smoke testing• Exploratory testing• Guerilla testing• Scenario testing• Installation testing• Load testing• System testingDon’t P

anic

And Lots of Questions Need To Be And Lots of Questions Need To Be

AnsweredAnswered

• 产品的组织结构图 产品测试后， 测试人员如何流动• 测试人员的职位和角色的划分 有没有独立的测试设计团队• 测试组织内部的回报关系， 测试人员如何考核 开发和测试的比例• 测试人员的要求，资历，经验等。 质量部对测试团队的监控作用。• 开发流程和开发模式 开发人员给测试人员的质量的保证• 测试工作量如何来评估，如何制定测试计划，依据什么来估计工作量• 开发和测试人员在各个阶段的工作职责测试人员是否参与百合子和黑盒子的测试• 版本发布谁来说了算，如何评估版本的质量。 版本能不能发布要考虑那方面的因素。• 工具和自动化， 仔细介绍一下好的自动化工具。有那些工具。日常测试中用那些工具。是否

自己开发的，还是商用工具。• 测试技术用例设计考虑的输入边界值 有否测试类型的划分• 测试用例设计的步骤，需求分析，测试设计从那些角度来考虑，是从用户需求还是从哪些。 • 测试策略，一个版本的测试策略，有老特性和新特性。 • 如何来自动化，进入维护状态，测试如何进行。 什么组织来负责。是否是独立的组织，人员

的交替关系。• 平台的测试和产品的测试的关系，是否有不同的测试团队来负责。测试内容和范围有没有差别• 一个版本，测试是否基于风险，需要考虑那些因素主要问题是在什么阶段发现的。转测试之前

还是之后。• 开发人员， SE ，市场人员和服务人员在测试过程中能发挥什么作用

3

Nature of the SoftwareNature of the Software• Continuous Release

4

Product QualityProduct Quality• “Rock Solid Product”, what does it mean?

• Your product is only as good as customer’s saying

• The ultimate goal is to increase customer satisfaction.

• C company publish a customer satisfaction index every day on its internal website.

• Customer are not satisfied when product is not function as they expected to be.

5

Why Not Satisfied?Why Not Satisfied?• Lack of product knowledge – need a TAC, not

the topic today ， and not quality problem. • Product has problem

o Manufacture problem – not the topic todayo Defects – yes, we need to catch those.

6

Internal Found Defects

Customer Found Defects

Product Quality Product Quality MetricsMetrics

• Product complexity• Numbers of the customer found defects• Customer found defects priority• Number of the unit shipped

Can you come up with an equation to calculate the “Product Quality Index” number?

You boss will love to say:” Lets reduce our quality index from 4.7 to 3.5 this year”.

7

QA’s goal QA’s goal • Reduce customer found defects!

• Reduce high profile customer found defects first. (80 percent of the customer only use 20 percent of the features, 20 percent of the customer make 80 percent of your revenue)

• So QA strategy is just anther ROI (return on investment)

8

2 Kinds of Testing 2 Kinds of Testing OrganizationOrganization

• By Feature

• By Testing roles

9

2 Types of Defects - 2 Types of Defects - EngineeringEngineering

• New feature defects – new code broken• Regression defects – new code break the old

codeo Enable new codeo Not enable new code

10

1Type of Defects - 1Type of Defects - CustomerCustomer

• Feature interaction defects

11

Why Regression Bug?Why Regression Bug?• Software Modules and Modules are coupled,

change one module’s code will affect others.

• Network equipments process same packet in different place when a packet traverse the box – sequential processing

• Protocols are layered (ISO layer)

• Protocols depends on one another (one protocol use another protocol as input)

12

Analysis Your Customer Analysis Your Customer Found BugsFound Bugs

• New feature defects (single or multi-feature interaction)o More testing need to be done on new feature testing.o Your testers need to be trained on how design

effective test cases to catch new bugs. o Different testing methodologies. o Increase your new feature testing coverage on your

test plan. o End-to-end customer oriented solution testing

• Regression defectso Will not decrease no matter how good your testers

areo You need a regression strategy o All code need to be tested whenever release a new

version ( huge work) 13

Combination is endlessCombination is endless

• Feature interaction is endless• The way to use your product/system is endless• Lucky you if you can cover them all• If not, you need a strategy• The best strategy is customer’s usage (20/80

rules)

14

To Reduce Customer To Reduce Customer Found DefectsFound Defects

• You need a process to test new• You need a process to test old• You need a process to test solution (whatever

on the customers site).

15

Bugs in a Product’s lifeBugs in a Product’s life

16

New Feature Bug

Regression Bug Regression Bug

Active Development, Market Share

Cash Cow End of Life

Product Revenue

Maintenance, feature enhancement

1.0

2.0

3.0

4.0 5.0

6.0

Nowadays in Company CNowadays in Company C

• Most of the new feature bugs can be found in the early development – strong testers, end-to-end system testing, early customer field trial.

• Many customer found defects are regression bugso 10 million lines of code need to be tested whenever a new version

is releasedo Release 2 new versions every week. o Is that possible? Yes, but only via Automation

• Many companies use frequent regression to stabilize a code branch.

17

Strategy To Release a Strategy To Release a Quality SoftwareQuality Software

• We are talking about carrier grade product.

• Test case/test plan design needs to start when code writing start, and finish when code writing finish.

• Be very strict on review process. (C company spent multi-million $$ on this)

• Closely monitor your bug trend during testing new.

• Repetitive regression to stabilize your code branch

18

Repetitive regression to stabilize your code Repetitive regression to stabilize your code

branch branch

• In order to fix regression bugs, new code need to be added introduce another regression bugs?

• Multiple regression cycles are needed to carve off regression bus.

• The more regression you are able to do before FCS, the more stabilize your code branch.

19

Testing Release Timing Testing Release Timing

20

Dev Test Regression1 Regression2 Regression3

Solution EFT

Golden

Week

Development and Testing At a GlanceDevelopment and Testing At a Glance

21

MRD/PRDFunctionalSpecification

CodeDevelopment Bug Fixes

Test Plan Test Case

CodeReleaseto Test

Daily/Frequent Code Drops

Code toTest

AlphaStart

BetaStart FCS

Various Testing Scripting

Dev

Test

CodeControl Opened Bug ID Code

Review

Form

GoldenWeek

No Need to Be In Sync No Need to Be In Sync

• Make regression a train• Make System testing and solution a train as

you like. • Only new feature testing need to be in sync

with development and release schedule. • Scheduled regression testing make branch

stable• Scheduled system and solution testing make

optimize your code for performance

22

Branch is QA’s Baby Branch is QA’s Baby

• Code branch should be owned by QA. Developer’s checking in new code need QA permission

• Design a nightly smoke test suite to monitor the build quality.

• Test new, and exploratory test new to take the new code bug out (daily build is necessary)

• Close the code branch before final regression. • Only regression commit is permitted during

regression testing• Multiple regression cycle to shrug off the

regression bug

23

Automate the Process to Make Your Life Automate the Process to Make Your Life

Easy Easy • When developer check in a code, he can specify a bug id. Then this bug in the bug tracking system’s state automatically change to “R”

• A new code commit automatically trigger a smoke testing, report automatically sent to QA team. If there are large amount of failures, an alarm can be automatically triggered (send a SMS to your cell phone?)

• ………

24

Not Enough Time Not Enough Time Testing?Testing?

•  More Attributes on the test cases can help

• Which functionality is most important to the project’s intended purpose?- Which functionality is most visible to the user?- Which functionality has the largest safety impact?- Which functionality has the largest financial impact on users?- Which aspects of the application are most important to the customer?- Which aspects of the application can be tested early in the development cycle?- Which parts of the code are most complex, and thus most subject to errors?- Which parts of the application were developed in rush or panic mode?- Which aspects of similar/related previous projects caused problems?- Which aspects of similar/related previous projects had large maintenance expenses?- Which parts of the requirements and design are unclear or poorly thought out?- What do the developers think are the highest-risk aspects of the application?- What kinds of problems would cause the worst publicity?- What kinds of problems would cause the most customer service complaints?- What kinds of tests could easily cover multiple functionalities?- Which tests will have the best high-risk-coverage to time-required ratio?

25

Ready to ReleaseReady to Release• QA is always asked: can we release it?

o QA’s job is to deliver report (a risk assessment), what works, what not. Testing report & Bug Report

• QA can only answer this question by giving a full version testing report

• QA use defects to talk to other department and bosses.

26

Big PictureBig Picture

27

Testing TipsTesting Tips• Plan ahead

o Perform good interview to create strategy and methodology

• Discussion is crucial:o Interview & review with right set of people: developers, marketing,

sales, manufacturero Use white board as much as possible to discuss new test cases,

resolving technical questions

• Work with developerso Find out the changes and possible breakageso Resolve ambiguities in varies documentations

28

Testing TipsTesting Tips• Testing

o Test the code changes and its dependencieso GUI testing != functionality testing. If possible, do not use

GUI testing to test core functionalitieso Use 80/20 rules to focus on the right area to testo Not only ask what to test, but ask what not to test

• Remove unnecessary duplicate efforts

o Types of release can determine the testing, see next slideo Weapon: Schedule. Use it carefullyo Code scale linearly but the complexity grows exponentially

29

Version controlVersion control• Version 1.0, Oct 10th, 2007, Liang Gao

• Version 1.1, Oct 29th, 2007, Liang Gao

• Version 1.3, Jan 10th, 2008, Liang Gao

• Version 1.4, March 13th, 2008, Liang Gao

• Version 1.5 June, 2008, Liang Gao

30