Upload
indicthreads
View
560
Download
1
Tags:
Embed Size (px)
Citation preview
Agenda Background and Motivation DFT in VLSI and hardware design Issues in Software Testing DFT for Software Development Automation and Testability Some Examples Cons, Review
VLSI/PCB Testing Issues Mass produced Each piece needs to be validated Complexity
− Number of subsystems− Amount of logic
Access to internal logic Testing costs Cost of recall
DFT Principles Controllability Observability
− Test Point Insertion Built-In Self Test (BIST) Fault Modelling Fault Simulation Test Pattern Generation
Software Testing“Testability is a design issue and needs to be
addressed with the design of the rest of the system. Projects that defer testing to the later stages of a project will find their programmers unwilling to consider testability changes by the time testers actually roll onto the project and start making suggestions. Testability takes cooperation, appreciation and a team commitment to reliability.”
- Bret Pettichord
Cost Of Bugs
10-25x10x1xCoding
25-100x15x10x1xArchitecture
10x-100x10x5-10x3x1xRequirements
ProductionSystem TestCodingArchitectureRequirements
Where Found
Where Introduced
Software Testability Suitability: clarity of specifications QA has Observability: we can only test what is visible Simplicity: easier design/UI makes testing easier Controllability: better the control, better the coverage
and automation Stability: how often it changes Performance: how fast it works Diagnosability: writing effective bug reports!
Software DFT Patterns BIST
− Automation suite w/ oracle
− Unit Tests− Assertions
Fault Simulation− Mock/Stub− Invalid params
Controllability− Decoupling− Bypassing− Mock/Stub− Set/Reset
Observability− Logging− Reporting− Test Interfaces
Good Automated Tests are Repeatable Easy to Write Easy to Understand Fast Way Easier with a Testable Software
When/How to Automate? Manager & Team are commited Testers/Dev either experienced or interested
in learning to script Product release cycles managed well Functionality / UI changes under control Early/Incremental approaches work best Build integration, reporting Each bug results in a automated test
Automation As Coding Automation scripts and frameworks are CODE! Use tools and scripting languages Evaluate tools on real problems Follow development processes
− Spiral/Agile Use Source Control same as rest of code These will have bugs as well!
Test “oracle” Expected Result
− Generated once from previous run or− Manually specified or− Legacy system
Time adjustment Results
− Database− File− Inline with code
Unit Tests Each module has independent set Developer written/maintained setup, teardown xUnit, TestNG Runs on build Can run sub-set on install TDD
Diagnosability All environments: Test / Staging / Production Errors result in visual messages Errors are raised where they occur Errors can be localized Details are sufficient to fix issue Ability to send diag data to dev
Logging Module Log levels (VERBOSE, INFO, WARNING, ERROR, CRITICAL, TIME) Define CRITICAL/ERROR levels well and use them! These should result in urgent notifications Lower levels in Production Time, App, Component, ThreadId, Message
8/2/2010 14:22pm: Notifier: Pop3: 8242: Sending welcome email to userid 334
Delimited columns for import Rotation based on Time, Size
Built-in Self Test Inserting test code/interfaces Set/Reset to bring state to known value Reporting to get current state Assertions about values/state
assert(order.billed==true)
“oracle” based regression
Some Examples Web Application
− Software As A Service− Browser Only UI− Ajax / Dynamic HTML
Banking (Client-Server)− Installed Application− Desktop Client− Server API
Web Application HTML Page Title Id values for important divs/controls Hidden values (non-textual, graphs, tables) Measurable Ajax responses Tools: Selenium, Sahi, Watir, WebTest, curl Logging incl. browser/ip/session cookie Ability to simulate time zone, language
Web Application - 2 Bypassing Captcha Mock TP APIs eg. Facebook, Google, OpenId SSL bypassing Transactions to be voided Multiple runs Always initialise controls
− Browser autofill
Desktop Unique identifiers to GUI controls Key Replay tools, SendKeys API test suites bypassing UI Client logs, ability to email Ability to run db queries in scripts
Desktop - 2 Automation Friendly Third Party Controls
− Ability to select cell− Copy-enable text fields− Key shortcuts to Forms UI for control focus,
clicking, navigation− Access by value (tree/list controls)
OCR for images/text consoles
Bypassing Credit card payment
− Test cards− Mock object− Dev auto-approval bypass code
Order placement− Dummy users, auto-fulfill− Dummy vendors (email order)− Admin screens move
Dates Never use system date directly Config override
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<appSettings>
<add key=“SystemDate" value=“$now"\>
</appSettings>
</configuration>
Nant: XMLPoke<xmlpoke file="App.config"
xpath="/configuration/appSettings/add[@key = 'SystemDate']/@value"
value="1-1-2009" />
Sending Emails Email subsystem should be a stub in dev Bypass SMTP: log to file … Email should be sent only to a whitelist Use Gmail Id & Atom API Use Unique Id/Timestamp to distinguish
Mocking / Stubbing Replace module during testing Different kinds:
− Sub-system (Email, Print)− Webservice (Credit Card, Flight Booking)− Hardware device (Biometric, CNC Machine)
Return default values, implement simple/fixed logic
Mocks - 2 Simulate Errors and Exceptions Provide more logging Jmock, Dependency Injection Con:
− Code level− Needs to change with API
Anti-Testability Viewpoints Security compromise by testability interfaces,
loggingRemove or lock down in prod
Extra coding timeMuch lower testing costs, better quality
Privacy issues in loggingShow only partial data
Performance Almost never an issue in practice!