Chasing Quality In Cloud Computing - Testing Different Levels Of Quality Requirements' by Kees Blokland

logo van Flair17-12-2010Polteq logo_RGB.png

Chasing Quality In Cloud Computing

Testing Different Levels Of Quality Requirements

Kees Blokland

[email protected] Polteq Testing Services BV, The Netherlands

Download recent version from www.polteq.com


2

Going to the cloud…

Test

managemen

t

applications

CRMERP

healthfinance

(test) environments

email

storage


3

bandwidth

Going to the cloud…

Test

managemen

t

applications

CRMERP

healthfinance

(test) environments

email

storage

virtualizationENABLERS

SOAstandard software

internet


4

Deployment models– private cloud– community cloud– public cloud– hybrid cloud

Service Models

Cloud Computing according to NIST

Essential characteristics

On-demand service

Broad network access

Resource pooling

Rapid elasticity

Measured service

Software as a Service

Platform as a Service

Infrastructure as a Service

US: National Institute of Standards and Technology http://www.nist.gov


5

Cloud Computing: risks and requirements

Essential characteristics On-demand serviceBroad network accessResource poolingRapid elasticityMeasured service

Deployment models– private cloud– community cloud– public cloud– hybrid cloud

Service Models

SaaS – Software as a Service

PaaS – Platform as a Service

IaaS – Infrastructure as a Service

SaaSPaaSIaaS

Security?

Performance? Legislation?

Privacy?

Vendor lock-in?

Elasticity?

Testability?

Multi platform?

User experience?

Migration? Continuity?

Integration?


6

From risk to test

Risk groups Test groups

PerformanceSecurityContinuityFunctionalityMaintainabilityLegislation and regulationsSuppliers…

PerformanceSecurityContinuityMigrationFunctionalityMaintainability Legislation End-to-endSelectionImplementationOperation…


7

Risk Groups – so far




8

Risk group: performance

• Response times too long– insufficient concurrent users– at (un)expected peaks

• Scalability, elasticity not working• Latency too high• Bandwidth, throughput too low• Up/download speed insufficient

! Other customers! Over-book, subscription model! Slow internet connection

On-demand service


Resource pooling

Rapid

elasticity

Measured service


9

Risk group: security

• Unauthorized access– administrators cloud service supplier– authorization/authentication inadequate– cyber crime, hackers, authorities– into cloud equipment building– ‘somewhere’ on the connection

• Data integrity– erased, not erased– unusable (loss of decryption key)

! Insecure internet connection! Insufficient data separation in equipment! Bring Your Own, insecure behavior users

On-demand service


Resource pooling


10

Risk group: continuity

• Cloud service unavailable– % availability is not achieved– supplier bankrupt or a conflict– internet connection lost

• Fall back plan does not work

! Internet connection malfunction! Other suppliers disturb the service! Supplier redundancy failure! Business instability supplier

On-demand service


Resource pooling

Rapid

elasticity

Measured service


11

• No fit on the business process• Low score on user friendliness• Not accessible everywhere• Not all mobile devices are supported• The equipment/configuration is not well performed• Customization is not well built• Integration with other systems fails

! Limitations in the Cloud Service! Bring Your Own Device, New Ways of Working! The evil Internet

Risk group: functionality

On-demand service



12

Risk group: maintainability

• Cloud service not testable• Manuals are inadequate because of changes• An end-to-end test is not possible• Unclear who is to solve problems• Cloud service not adaptable to new requirements

! Cloud service changes unannounced! Cloud service not configurable! No test environment for cloud service! No helpdesk

On-demand service

How to keep it up and running?


13

Risk group: legislation and regulations

• Violating EU data protection directive– location, security data– ownership, agreements with data processors

• Violating EU data retention directive• Bankruptcy of supplier inhibits keeping obligations• No grip on what happens to data

– warrant in other country

! Where are my data?! Conflicting or unclear legislation! Role of (unreliable) authorities

US: Patriot Act

On-demand service


Rapid

elasticity


14

Risk group: supplier

• Bankruptcy, conflict• At the mercy of the supplier

– (pay-per-use) conditions change– cloud service changes

• Quality not stable, unreliable• Difficult to switch

– to another supplier– back

! Vendor lock-in, powerful supplier! No insight in quality SW development! Developments (technology, growth, take-overs, …)

There is no fit any more

Supplier unmasked


15

Test Groups – so far




16

Test Groups – so far


PerformanceSecurityContinuityMigrationFunctionalityMaintainability Legislation End-to-endSelectionImplementationOperation…End-to-end testing

Operational profilesTesting of PackagesBVA-STT-DCoT-DCT

Load & Stress


17

Test group: performance

• What are the acceptance criteria?• Load testing• Stress testing

– not always allowed– what happens at the boundaries of the “bundle”

• Endurance test, volume test– restricted possibilities: fair use policy– monitors

• Elasticity, pay-per-use– LOAD+PCT+BVA


18

Test group: performance

• Test cases based on load profiles

• Load profiles based on operational profiles

• Test environment = production environment

• Testing in real time– under operating conditions– with the “cloud shop open”


19

Testing Elasticity

100

usage

time

Load profile – ‘UP’

99

100

101

Boundary values ‘UP’ Load profile – ‘DOWN’

Boundary values ‘UP’test case 1: usage=99, paid for 100test case 2: usage=100, paid for 100test case 3: usage=101, paid for 200

Boundary values ‘DOWN’test case 1: usage=101, paid for 200test case 2: usage=100, paid for 100test case 3: usage=99, paid for 100

200

max=100max=100

wantextension?

max=200200 billed

max=100100 billed

no

yes

Process Cycle Test


20

Test group: security

• Make inventory of security measures– Internet connection

– Cloud service

– Client

http/ssl vpnwifi/wap data encryption

login identity management autorisation profile

access to building logs

IDaaS

weak passwords

authorisation

pincode mobiles

door closed patch routine

patch routine

social engineering

firewall

firewall

Security measuresAuthorisationAuthenticationTechnical facilities Security updatesBehaviour of peopleLogging


21

Test group: security

• Testing and assessing– Assessing end-to-end security architecture

– Functional tests

– Tests by specialists

authorisation authentication encryption logs

encryption technique authentication technique

technical infrastructure

physical security

data separation

audit trailspatch update routine

hackers test audit

Specialists n

eeded

Specialists n

eeded


22

Test group: continuity

• Testing of redundancy, fall back• Off line• Continuous end-to-end regression test

• Measuring the availability– 99.99….9%– critical moments– MTBF, MTTR

• What-if scenarios– disaster recovery– internet unavailable– …

Fail over testing with State Transition Test


23

Test group: migration

• Where goes the data?• To/from/between cloud services

• Data repair: testing data• Testing the data conversion tool

• Data conversion– checklist – performance– security

! cloud service is

tested

! detailed planning

! sufficient tim

e

! technical knowledge

CHECKLIST MIGRATIONminimal disruptionno data lossconversion successfullyno hanging transactionsno loss due to bad data…


24

• Testing SaaS = testing of standard software package• Testing:

– fit between cloud service and business process– configuration of the cloud service– integration of cloud service with other systems– multi client platforms– the end-to-end business process

• What is the test basis?– the old system– process descriptions, use cases– (functional) operational profiles

Test group: functionality

ClassificationTrees

ProcessModels


25

Test group: maintainability

• Test environments– Public: none, stubs & mocks– Private: to be negotiated

• Manuals– Public: instructions for use– Private: custom manuals, also for maintenance

• Change procedure– Public: announcements supplier– Private: to be negotiated

• Helpdesk– Incident handling


26

Test group: legislation and regulation

• Storage and processing of data– examples…

• Influence of the authorities– examples…

• How is the test manager supposed to deal with it?– ensure that it is taken into account– ensure that lawyers are involved– bridge between ICT and lawyer

what is the ris

k of non-compliancy?

example: who does n

ot use production data for testin

g?


27

Broad role of the Test Manager

Implementation testing, testing, testing


28


Implementation: what to test?

Risk groups

Test groupsPerformanceSecurityContinuityFunctionalityMaintainabilityLegislation and regulationsSuppliers…

Cloud Service selected!


29


Selection

Implementation

risks, criteria, advice, contract

testing, testing, testing


30

Selection: the risks

PublicSaaSPublicSaaS

Intention: introducing Cloud Computing

Cloud Risks


31

Selection: criteria

Intention: introducing Cloud Computing

Selection criteria

Cost reductionBusiness processPerformanceScalabilityNew ways of workingContinuityMigrationSecurityIntegration…


32


Selection

Implementation

Operation

risks, criteria, advice, contract

testing, testing, testing

end-to-end regression test, evaluation


33

Operation: everything is moving

OperationOperation

changes in other systems

internet changes

changes in clients

changes in

business

process

changes in cloud service

changes cloud supplier

Release Calendar?Change Process?

Continuous End-to-end Test

growth


34

Operation, role of the test manager

• Make inventory of cloud continuity risks– everything is moving!

• Periodic end-to-end testing– is it still working?

…- end-to-end-to-end-to-end-end-to-end-to-end-to-end-to-…


35

Cloud & perspective of testing

From Risk To Test

Everything is moving

Broad Role Test Manager

End to End and the rest


Questions?


Thank you!