Solving the Software Quality Puzzle www.psqtconference.com
How We Test @ MicrosoftHow We Test @ Microsoft
How Microsoft Tests Software Today How Microsoft Tests Software Today and Where We are Headed Nextand Where We are Headed Next
Ken Johnston, Director Test ExcellenceMicrosoft Engineering ExcellencePSQT North 2006
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Making the hard callMaking the hard call
Sev 1 BugSev 1 BugCrashes setup and Crashes setup and leaves the machine in leaves the machine in an unknown statean unknown stateOn upgrade install it On upgrade install it corrupts customer datacorrupts customer data
Reported by Reported by customercustomerReported by Dev, Reported by Dev, Test and PMTest and PMLast bug left!Last bug left!
What’s a young test What’s a young test lead to do?lead to do?
Sasquatch
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Global CompanyGlobal Company
Multinational workforceMultinational workforceWorldwide customers and Worldwide customers and
partnerspartners
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Products in Different CategoriesProducts in Different Categories
260 Products 260 Products Released in last Released in last 12 Months12 Months
There is no 1 There is no 1
““Microsoft Way”Microsoft Way”
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
There is No One Way to ShipThere is No One Way to ShipMany different ship cyclesMany different ship cycles– Daily/WeeklyDaily/Weekly– MonthlyMonthly– AnnuallyAnnually– Every 2-3 yearsEvery 2-3 years
• More than one product has followed this pathMore than one product has followed this path• Multiple Slip ScheduleMultiple Slip Schedule
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Many Development Methods UsedMany Development Methods Used
Spiral Iterative/WaterfallSpiral Iterative/Waterfall
Agile TechniquesAgile TechniquesTest Driven DevelopmentTest Driven DevelopmentSCRUMSCRUM
TSP/PSPTSP/PSPTeam Software Plan & Personal Software PlanTeam Software Plan & Personal Software Plan
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
What to do with 50 min left?What to do with 50 min left?Look at the people landscapeLook at the people landscape
Cover as many techniques and tools as I canCover as many techniques and tools as I can
Try to squeeze in a little on the future of Try to squeeze in a little on the future of testingtesting
At the end I will point you to additional At the end I will point you to additional resourcesresources
With any luck we’ll have some fun along the With any luck we’ll have some fun along the wayway
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
People, Process and ToolsPeople, Process and Tools
Testing @Testing @
MicrosoftMicrosoft
PeoplePeople ProcesProcesss
ToolsTools
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
How many Engineers are There?How many Engineers are There?
More than More than 31,00031,000 Engineers World Wide Engineers World Wide
More than More than 7,5007,500 Test Engineers Test Engineers
What is the dev to test ratio?What is the dev to test ratio?Counting full time and temp/vendor Test Engineers Counting full time and temp/vendor Test Engineers Microsoft has more than 1 Test Engineer for every Microsoft has more than 1 Test Engineer for every DeveloperDeveloper
Hired more than Hired more than 4,0004,000 Engineers FY06 Engineers FY06Hired more than Hired more than 1,000 1,000 SDETs in FY06SDETs in FY06
Too many open positionsToo many open positionsMore than More than 5,000 5,000 open Engineering positionsopen Engineering positionsMore than More than 1,0001,000 open Test Engineer positionsopen Test Engineer positions
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
SDET ProfileSDET Profile
Mostly Computer Science Graduates & Industry Mostly Computer Science Graduates & Industry HiresHiresAlsoAlso
Electrical EngineeringElectrical EngineeringMathMathPhysicsPhysicsOther EngineeringOther Engineering
We look forWe look forStrong Programming SkillsStrong Programming SkillsProblem Decomposition SkillsProblem Decomposition SkillsDrive for ResultsDrive for ResultsPassion for QualityPassion for Quality
Experience in Test is not the highest order bitExperience in Test is not the highest order bit
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Test Techniques Taught in New Test Techniques Taught in New TesterTester
Functional testing Functional testing techniquestechniques
Boundary value analysisBoundary value analysisEquivalence class Equivalence class partitioningpartitioningCombinatorial analysisCombinatorial analysis
Structural testing Structural testing techniques and code techniques and code coverage analysiscoverage analysis
Decision/Branch Decision/Branch coveragecoverageCondition coverageCondition coverageBasis path coverageBasis path coverageCode coverage analysisCode coverage analysis
Integration and system Integration and system level testing level testing
Exploratory testingExploratory testingApplication compatibility Application compatibility testingtestingAccessibility testingAccessibility testingSecurity testingSecurity testingGlobalization testingGlobalization testing
DebuggingDebugging Inspections and ReviewInspections and Review Model Based TestingModel Based Testing
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
DevelopmentDevelopment
TestTest
The Feature Team TriadThe Feature Team Triad
Program Program
ManagementManagement
Simple DescriptionsSimple Descriptions
Program Managers (PMs) Program Managers (PMs) research customer needs, research customer needs, competitors and develop the competitors and develop the requirementsrequirements
Developers (SDEs)write the Developers (SDEs)write the product code to the product code to the functional requirementsfunctional requirements
Test Engineers (SDETs) Test Engineers (SDETs) ensures the product meets ensures the product meets the functional requirementsthe functional requirements
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
DevelopmentDevelopment
TestTest
The Quartet for Web ServicesThe Quartet for Web Services
Program Program
ManagementManagement
IT/OperationsIT/Operations
Operations is critical Operations is critical partner in migration to partner in migration to web servicesweb services
~4,500 IT/Ops Engineers~4,500 IT/Ops Engineers
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Many More Engineering Many More Engineering DisciplinesDisciplines
DevelopmentDevelopment
TestTest
Program Program
ManagementManagement
IT/OperationsIT/Operations
IPEIPE UsabilityUsability
DesignDesignCreativeCreative
ContentContent ResearchResearchEngineeringEngineering
ManagementManagement
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Focus on a Few Processes and Focus on a Few Processes and TechniquesTechniques
Testing @Testing @
MicrosoftMicrosoft
PeoplePeople ProcesProcesss
ToolsTools
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Run test caseEnter bug
report
Modifydesign
No or refine
Designor code
Design issue
Close bug
Reactivatebug
Investigation approved
Sustained Engineering
Transition Checklist
Design Okay
Regressbug
Pass
Fail
Not approved
Proposed code fix
Proposed design fix
Code issue
Fix approved
ProductCode
Specify code fix
Fix codeResolve
fixed bugBuild
workflow
Triage
Test Execution
Status Report Closethe bug
Resolve "Won't Fix"
Bugs Drive WorkBugs Drive Work
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Not all Bugs are Code Not all Bugs are Code DefectsDefects
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Perform pre-build
(sync codeand tools)
Perform compile and
code analysis (PREfast)
Perform link and
binary analysis (PREfix)
Product Code
Conduct build verification
tests
Conduct check-in tests
(unit tests)
Conduct post-build (cleanup)
Clean CIT (image, setup,
test)
Private and/or Buddy Build
Self-Test Build
Self-Host Build
Create the checkpoint release (copy source and
components)
Create release build (copy source
and binaries)
The Daily BuildThe Daily Build
The Heartbeat of Microsoft R&D
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Build Reporting on Every Build Reporting on Every BuildBuild
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Too Many ToolsToo Many Tools
Testing @Testing @
MicrosoftMicrosoft
PeoplePeople ProcesProcesss
ToolsTools
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Microsoft has a lot of ToolsMicrosoft has a lot of Tools
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Active Customer Feedback
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Play Balance - Play Balance -
01a_look_start
01a_move_start
01a_shield_start
01a_tram_start
01b_1st_start
01b_1st_wv4
01b_1st_wv5_end
01b_atr2_start
01b_atr2_mid
01b_bay1_start
01b_bay1_mid
01b_bay2_start
01b_bay2_mid
01b_arm_start
01b_atr1_start
01b_trm1_start
01b_dck_start
01b_elv_start
01b_lvr_start
01b_gun_start
e1_rooftop_defense
e2_hunter_intro
e3_highway_underpass
e4_sniper_alley
e5_neighborhood
e6_hotel_entrance
e7_hotel_hall
e8_hotel_exit
e9_beach_infantry
e10_beach_fort
e11_tunnel_entrance
e12_tunnel_blockades
e13_tunnel_convoy
e14_bridge_ascent
e15_bridge_summit
e16_bridge_descent
e17_flooded_tunnel
e18_park
enc_ledge_lift
enc_ledge_lower
enc_high_hall_a
enc_high_hall_b
enc_room_a
enc_jail_ini
enc_jail_cell_a
enc_jail_cell_b
enc_jail_down_lift
enc_corridors_b
enc_tower_a_exterior
enc_gardens_a
enc_mid_tower
e4_generator_room
e2_scarab_cove
e19_park
e20_road_skirmish
e21_hospital_seige
e22_scarab_intro
e23_scarab_boarding
04a_1_landing_zone
04a_2_recycling_center
04a_3_hangar
04a_4_first_halls
04a_5_underhangar
04a_6_second_halls
04a_7_bottling_plant
04a_8_bottling_back
04a_9_dogfight
04b_1_entry
04b_2_disposal_chamber
04b_3_silo_start
04b_4_silo_mid
04b_5_silo_end
04b_6_hall_to_lab
04b_7_flood_lab_start
04b_8_flood_lab_mid_01
04b_9_flood_lab_mid_02
04b_10_flood_lab_end
04b_11_wraparound
04b_12_control_room_01
04b_13_control_room_02
04b_14_cable_room
04b_15_control_return
04b_16_power_core
04b_17_dogfight
04b_18_bottling_return_01
04b_19_bottling_return_02
04b_20_halls_to_under_01
04b_21_halls_to_under_02
04b_22_halls_to_under_03
04b_23_underhangar_return
04b_24_boss_fight_01
04b_25_boss_fight_02
04b_26_boss_fight_03
enc_maus_grand
enc_maus_bridge
enc_mausoleum
enc_maus_midpoint
e1_stealthy_insertion
e2_cylinder_room
e3_ramp_room0
e4_step_room
e5_ramp_room1
e6_ghost_ledge
e7_kidney0
e8_kidney1
e9_ski_run
04b_27_boss_fight_04
04b_28_boss_fight_05
04b_29_boss_fight_06
04b_30_boss_fight_07
05a_1_landing_zone
05a_2_overlook
05a_3_bridge
05a_4_winding_path
05a_5_temple_entrance
05a_6_tunnel
05a_7_old_temple
05a_8_old_temple_courtyard
05a_9_grotto
05a_10_tower_approach
05a_11_tower1_interior
05b_1_tower1
05b_2_bridges
05b_3_tower2
05b_4_central_platform
05b_5_gondola_01
05b_6_tower3
05b_7_tunnel1
05b_8_sunken_chamber
05b_9_tunnel2
05b_10_island_interior
05b_11_island_gully
05b_12_island_exterior
05b_13_gondola_02
05b_14_temple_exterior
05b_15_temple_halls
05b_16_temple_center
05b_17_temple_retreat
enc_insertion
enc_hall_a
enc_cond_a_sec_a
enc_cond_a_sec_b
enc_hall_b
enc_plug_launch
enc_plug_across
enc_plug_landing
e5_bridge
e6_jail
e7_scarab_ledge
e8_scarab_cove
e9_scarab_canyon
e11_scarab_boardings
e12_antechamber
e13_control_room
enc_gardens_b
enc_tower_b_exterior
e10_brute_camp
phantom_ledge
e3_ramp_room
enc_plug_landing_bk
enc_plug_landing_lower
enc_hall_c_sec_a
enc_hall_c_sec_b
enc_hall_c_sec_c
enc_ledge_a_sec_a
enc_ledge_a_sec_b
enc_cond_b_sec_a
enc_cond_b_sec_b
enc_cond_b_sec_c
enc_ledge_b
enc_ledge_c
enc_qz_initial_camp
enc_covenant_defense
enc_cov_charge
enc_vehicle_int
enc_vehicle_int_bk
enc_qz_ext_a_dam
enc_qz_ext_a
enc_ext_a_fact_ent
enc_crashed_factory_a
enc_crashed_factory_ext
enc_crashed_factory_b
enc_ext_b_fact_exit
enc_qz_ext_b
enc_qz_ext_b_bk
enc_qz_ext_b_exit
enc_e21
enc_e22
enc_e23
enc_e24
enc_e25
enc_council_wave_1
enc_council_pedestal
enc_council_wave_2
enc_jail_cell_a
enc_jail_cell_b
enc_council_ini
enc_grand_a
enc_ledge_ini
turbolift_base
plasma_chandelier
riptide_start
riptide_ambush
space_cave1
boobag_garden
space_cave2
hallway_attack
inner_sanctum
council_chamber
last_room
e1_wraith_canyon
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Halo 2 Playtest DataHalo 2 Playtest Data
0
2
4
6
8
10
12
enc_
cov_
char
ge
enc_
veh
icle
_in
t
enc_
veh
icle
_in
t_b
k
enc_
qz_
ext_
a_d
am
enc_
qz_
ext_
a
enc_
ext_
a_fa
ct_e
nt
enc_
cras
hed
_fac
tory
_a
enc_
cras
hed
_fac
tory
_ext
enc_
cras
hed
_fac
tory
_b
enc_
ext_
b_f
act_
exit
enc_
qz_
ext_
b
enc_
qz_
ext_
b_b
k
enc_
qz_
ext_
b_e
xit
enc_
e21
enc_
e23
enc_
e25
Median
Mean
“unknown” deaths
≈ 8+ hours in…
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Software Quality Metrics (SQM)Software Quality Metrics (SQM)
Software Quality Metrics / Software Quantitative Software Quality Metrics / Software Quantitative MechanicsMechanics
Client-side tool which measures statistical usage Client-side tool which measures statistical usage datadata
Server-side collection, reporting and management Server-side collection, reporting and management tooltool
Shared SAN Datawarehouse
Client SQMIntegrator
Front EndCluster
ReportingCluster
Dedicated Processing
DedicatedDatawarehouse
Shared Processing
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
O11 Beta2
O11 Beta 2TR
O11 RTM Early
O11 RTM All
O11 SP1 All
M1 Dogfood
M2 Dogfood
SQM in Office 2003 and Office SQM in Office 2003 and Office 1212
Mean
Tim
e T
o C
rash
(hours) 0
100
200
300
400
500
600
700
800
900
One Note OutlookApplication
Word
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Windows Error Reporting (a.k.a. Windows Error Reporting (a.k.a. Watson)Watson)
Real-world data about user painReal-world data about user pain– Millions of hits every dayMillions of hits every day
Walt Mossberg 9/30/99:Walt Mossberg 9/30/99: “ “Dear Diary: I’m tired of the way Windows crashes and freezes…”Dear Diary: I’m tired of the way Windows crashes and freezes…”
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Continuous Feedback ProcessContinuous Feedback Process
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Tests and Test Tools
Bug ManagementBug Management
Test Case ManagementTest Case Management
Automation FocusAutomation Focus
Static Code AnalysisStatic Code Analysis
Dynamic AnalysisDynamic Analysis
Unit TestsUnit Tests
Code CoverageCode Coverage
Performance/LoadPerformance/Load
Web Load TestingWeb Load Testing
Model Based TestingModel Based Testing
Threat ModelingThreat Modeling
Fuzz TestingFuzz Testing
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
For Bugs We Use Product For Bugs We Use Product StudioStudio
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Test Case ManagementTest Case Management
Test Case ManagementManual and Automated in one system
Test Case ManagementManual and Automated in one system
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Microsoft has an Automation Microsoft has an Automation FocusFocus
Windows Vista + Office 12 > 200 MLOCWindows Vista + Office 12 > 200 MLOC
Most products target ~70% code coverage before Most products target ~70% code coverage before shipship
Some as high as 80% on unit testsSome as high as 80% on unit tests
Microsoft has >110,000 Test MachinesMicrosoft has >110,000 Test MachinesFY07 lab budgets 100 million+FY07 lab budgets 100 million+
Windows Vista has over 6.7 Million automated test Windows Vista has over 6.7 Million automated test jobsjobs
Office 12 has over 1 Million automated test casesOffice 12 has over 1 Million automated test cases
Why Automate so Much?Why Automate so Much?Many products must be supported for 10 yearsMany products must be supported for 10 yearsCompatibility regression passesCompatibility regression passesDaily build process increases reuseDaily build process increases reuse
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Every SDET Learns to Every SDET Learns to useuseDebuggersDebuggers
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Static Code AnalysisStatic Code Analysis
Static Code AnalyzerStatic Code Analyzer
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Dynamic Analysis – Dynamic Analysis – AppVerifier and TruScanAppVerifier and TruScanMemory Leak CheckerMemory Leak Checker
Unutilized Variable CheckerUnutilized Variable Checker
Race Condition CheckerRace Condition Checker
Handle leak checkerHandle leak checker
Buffer Overrun checkerBuffer Overrun checker
Dangling Pointer checkerDangling Pointer checker
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Unit TestingUnit Testing
Integrated Unit TestingIntegrated Unit Testing
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Code CoverageCode Coverage
Code CoverageUnit, BVT, Suite, All
Code CoverageUnit, BVT, Suite, All
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Performance measures of test and Systems Under Test
Alerts and warnings on Systems Under Test
Load GenerationLoad Generation
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Timeline of memory consumption
Suspect functions, drillable to code
Code PerformanceCode Performance
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
View of content as rendered
Content validation
http request & response
Performance breakdown
Data substitution
Web TestingWeb Testing
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Model Based TestingModel Based Testing
Significant growth in Significant growth in use of Model Based use of Model Based TestingTesting
CoursesCoursesLecturesLecturesE-LearningE-LearningActive communityActive community
MDE – Model Design MDE – Model Design EnvironmentEnvironment
Spec ExplorerSpec Explorerhttp://research.microsoft.com/SpecExplorer/Built on Spec Sharp Built on Spec Sharp (extension to C#)(extension to C#)http://research.microsoft.com/specsharp
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Threat Modeling & Fuzz TestingThreat Modeling & Fuzz Testing
Threat Modeling - analyze the design for vulnerabilityThreat Modeling - analyze the design for vulnerabilityModel data flows Model data flows (How to Write Secure Code)(How to Write Secure Code)
SS - Spoofing Identity- Spoofing IdentityTT - Tampering with Data- Tampering with DataRR - Repudiation- RepudiationII - Information Disclosure- Information DisclosureDD - Denial of Service- Denial of ServiceEE - Elevation of Privilege- Elevation of Privilege
Fuzz testing is a method of finding software security holes by Fuzz testing is a method of finding software security holes by feeding purposely invalid and ill-formed data as input to program feeding purposely invalid and ill-formed data as input to program interfaces. Inputs include:interfaces. Inputs include:
FilesFilesNetwork portsNetwork portsAPIsAPIs
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
A Few Other QA Processes
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Global Software CompanyGlobal Software Company
Ship Products World WideShip Products World WideLocalize products in >50 LanguagesLocalize products in >50 LanguagesFocus of IPE (formerly Localization) disciplineFocus of IPE (formerly Localization) discipline
Globalization/Global SufficiencyGlobalization/Global SufficiencyLocalizationLocalizationMarketizationMarketizationGeopolitical Geopolitical
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
LocStudio – the Tool for IPELocStudio – the Tool for IPE
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Who’s Behind the One-way Who’s Behind the One-way Mirror?Mirror?
What do we get from Usability?What do we get from Usability?
provides perspective onto the provides perspective onto the
total situationtotal situation
reveals nuances of behavior, reveals nuances of behavior,
motivation, and emotion motivation, and emotion
enables comparison and enables comparison and
contrasting of attitudes, actions, contrasting of attitudes, actions,
and feelingsand feelings
uncovers latent needs users uncovers latent needs users
can’t articulatecan’t articulate
fosters user inspired empathy, fosters user inspired empathy,
advocacy, and creativityadvocacy, and creativity
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Usability Labs Without WallsUsability Labs Without Walls
Virtual Research LabVirtual Research Lab
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Where are We Headed Next?
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Metrics then, now, and…Metrics then, now, and…
Product Size 1995 Office 95
MLOC – ~4MMLOC – ~4MTC – UnkTC – UnkBugs – 268KBugs – 268K
Windows 95
MLOC – ~7MMLOC – ~7M
TC TC –– Unk UnkBugs – >5*25KBugs – >5*25K
Data Sources 2006
Product Studio Test Case P/FLarge Checkin Suites
Automated per build BVTsAutomated test case results
Automatic bug filing
Stress, Scale, Perf, Icecap
PSSGranular call coding
Customer impact and cost
Customer feedbacksystems
Watson, SQM
Static AnalysisPreFix, PreFast, FXCop
Dynamic AnalysisApp Verifier, Driver Verifier
Magellancode coverage,
code churn, code complexity
Product Size 2006 Office 12
Blocks – 19.1MBlocks – 19.1MTC – ~1MTC – ~1M
VistaBlocks – 34.7MBlocks – 34.7MWTT Jobs – 6.7MWTT Jobs – 6.7M
Fast forward 11 years
Data Sources 1995
RAIDTest Case
Simple DB
Word Docs
Excel Files
Stress, Scale, Perf
PSS
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Metrics next…Metrics next…Move from Quality Control to Move from Quality Control to Quality AssuranceQuality Assurance
QC is fixing the bugs we findQC is fixing the bugs we findQA is setting the goals for QA is setting the goals for key indicators that we know key indicators that we know will ensure high qualitywill ensure high quality
Data warehouse projects in Data warehouse projects in many product teamsmany product teams
PHI, Team Stats, MSN QA PHI, Team Stats, MSN QA Dashboard, MSN QOS, Dashboard, MSN QOS, Exchange, Office, Exchange, Office, Cross-correlation of data across Cross-correlation of data across large numbers of productslarge numbers of productsAuto-analysis of dataAuto-analysis of data
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Testability – SOCK Testability – SOCK SimplicitySimplicity & Consistency & Consistency
Reducing cost of testing at each phase of SEARCHReducing cost of testing at each phase of SEARCHLess to test, more quickly we can test.Less to test, more quickly we can test.DesignDesignSimilar parts behave in a similar mannerSimilar parts behave in a similar manner
Design & Test Patterns
Standards, e.g. GUI, Design Guidelines
Naming conventions
ObservabilityObservabilityExposing state (aka Visibility or Transparency)Exposing state (aka Visibility or Transparency)
ControlControlExercising every nook & cranny as efficiently as possibleExercising every nook & cranny as efficiently as possible
Knowledge of expected results (aka Documentation)Knowledge of expected results (aka Documentation)Is observed behavior correct?Is observed behavior correct?
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Getting Back into Code ChurnGetting Back into Code ChurnRecent research has proven Code Recent research has proven Code ChurnChurn
Windows Research showed correlation to Windows Research showed correlation to post ship defect densitypost ship defect density
Code Churn can be pulled fromCode Churn can be pulled fromSource Depot (source control)Source Depot (source control)Magellan (code coverage tool)Magellan (code coverage tool)
Not used to stop shipNot used to stop shipGood tool for risk indicationGood tool for risk indication
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Tempest: Consolidated Complexity Tempest: Consolidated Complexity MetricsMetricsMeasures failure proneness based on statistical analysis of Measures failure proneness based on statistical analysis of previous failuresprevious failures
Leverages size, complexity, and coupling metrics to model Leverages size, complexity, and coupling metrics to model failuresfailures
Gives confidence bounds for code metrics Gives confidence bounds for code metrics collected through the binary and pdbcollected through the binary and pdb
Scalable and runs on the desktopScalable and runs on the desktopComplete Vista build can be Complete Vista build can be analyzed in 1.25 hours.analyzed in 1.25 hours.
Internal tool only right nowInternal tool only right now
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Auto AnalysisAuto Analysis
110,000 Test 110,000 Test MachinesMachines
Millions of Millions of Automated Test Automated Test CasesCases
>500 Builds a day>500 Builds a dayThe volume of defects keeps increasingThe volume of defects keeps increasing
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
ResourcesResourcesAbout Testers (SDETs) at MicrosoftAbout Testers (SDETs) at Microsoft
Recruiting - Recruiting - http://www.http://www.microsoftmicrosoft.com/college/ft_.com/college/ft_softdesengtestsoftdesengtest..mspxmspx
http://members.http://members.microsoftmicrosoft.com/careers/.com/careers/careerpathcareerpath/technical//technical/softwaretestingsoftwaretesting..mspxmspx
Public Research Papers - Public Research Papers - http://research.http://research.microsoftmicrosoft.com.com
Some of the Tools from this presentationSome of the Tools from this presentation
•• Visual Studio - Visual Studio - http://http://msdnmsdn..microsoftmicrosoft.com/.com/vstudiovstudio//teamsystemteamsystem/tester//tester/
http://msdn2.http://msdn2.microsoftmicrosoft.com/en-us/default..com/en-us/default.aspxaspx
AppVerifier AppVerifier –– http://www.http://www.microsoftmicrosoft.com/windows/.com/windows/appexperienceappexperience
FXCOP – FXCOP – http://www.http://www.gotdotnetgotdotnet.com/team/.com/team/fxcopfxcop//
Blogs and Community ResourcesBlogs and Community Resources
Wiki content and video profiles - Wiki content and video profiles - http://channel9.http://channel9.msdnmsdn.com/.com/Bug Triage Video - Bug Triage Video - http://channel9.msdn.com/ShowPost.aspx?PostID=26792#26792 http://channel9.msdn.com/ShowPost.aspx?PostID=26792#26792
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Ken Johnston, Director Test ExcellenceMicrosoft Engineering [email protected]
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
ResourcesResources
Research Papers (Nachi Nagappan)Research Papers (Nachi Nagappan)http://teamv2/sites/tvm http://teamv2/sites/tvm
Consolidated reportingConsolidated reporting
http://teamstatshttp://teamstats
http://phihttp://phi
http://office/c16/metricshttp://office/c16/metrics
This slide deck and related resources:This slide deck and related resources:http://eetwcforumhttp://eetwcforum
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
FxCop ResourcesFxCop Resourceshttp://www.gotdotnet.com/team/fxcop/http://www.gotdotnet.com/team/fxcop/
FxCop download site/bulletin boardFxCop download site/bulletin boardFxCop docs and rule topicsFxCop docs and rule topicsSupportSupport
[email protected]@Microsoft.com
AppVerifier ResourcesAppVerifier ResourcesDownload Site: Download Site: http://www.microsoft.com/windows/appexperiencehttp://www.microsoft.com/windows/appexperience
Newsgroup:Newsgroup:microsoft.public.win32.programmer.toolsmicrosoft.public.win32.programmer.tools http://msdn.microsoft.com/newsgroups/managedhttp://msdn.microsoft.com/newsgroups/managed
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
General Security ResourcesGeneral Security ResourcesGeneral General
http://www.microsoft.com/security http://www.microsoft.com/security
XP SP2 Resources for the IT Professional XP SP2 Resources for the IT Professional http://www.microsoft.com/technet/winxpsp2 http://www.microsoft.com/technet/winxpsp2
Security Guidance Center Security Guidance Center http://www.microsoft.com/security/guidance http://www.microsoft.com/security/guidance
Tools Tools http://www.microsoft.com/technet/Security/tools http://www.microsoft.com/technet/Security/tools
How Microsoft IT Secures Microsoft How Microsoft IT Secures Microsoft http://www.microsoft.com/technet/itsolutions/msit http://www.microsoft.com/technet/itsolutions/msit
E-Learning Clinics E-Learning Clinics https://www.microsoftelearning.com/security https://www.microsoftelearning.com/security
Events and Webcasts Events and Webcasts http://www.microsoft.com/seminar/events/security.mspx http://www.microsoft.com/seminar/events/security.mspx
Microsoft Engineering Excellence
Solving the Software Quality Puzzle www.psqtconference.com
Additional DocumentationAdditional Documentation
Writing Secure Code: Writing Secure Code: http://www.microsoft.com/MSPress/books/5957.asp http://www.microsoft.com/MSPress/books/5957.asp
Thread Modeling: Thread Modeling: http://www.microsoft.com/MSPress/books/6892.asphttp://www.microsoft.com/MSPress/books/6892.asp
Threat Modeling OnlineThreat Modeling Onlinehttp://msdn.microsoft.com/security/securecode/http://msdn.microsoft.com/security/securecode/
threatmodeling/default.aspxthreatmodeling/default.aspx Compiler Security Checks In Depth Compiler Security Checks In Depth http://msdn.microsoft.com/library/default.asp?url=/library/en-http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dv_vstechart/html/vctchcompilersecuritychecksindepth.aspus/dv_vstechart/html/vctchcompilersecuritychecksindepth.asp