Model-Based Testing: Concepts, Tools, and Techniques

  • Published on
    04-Dec-2014

  • View
    206

  • Download
    2

Embed Size (px)

DESCRIPTION

For decades, software development tools and methods have evolved with an emphasis on modeling. Standards like UML and SysML are now used to develop some of the most complex systems in the world. However, test design remains a largely manual, intuitive process. Now, a significant opportunity exists for testing organizations to realize the benefits of modeling. Adam Richards describes how to leverage model-based testing to dramatically improve both test coverage and efficiencyand lower the overall cost of quality. Adam provides an overview of the basic concepts and process implications of model-based testing, including its role in agile. A survey of model types and techniques shows different model-based solutions for different kinds of testing problems. Explore tool integrations and weigh the pros and cons of model-based test development against a variety of system and project-level factors. Gain a working knowledge of the concepts, tools, and techniques needed to introduce model-based testing to your organization.

Transcript

  • 1. W14 Concurrent Class 10/2/2013 3:00:00 PM "Model-Based Testing: Concepts, Tools, and Techniques" Presented by: Adam Richards Critical Logic Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 904-278-0524 sqeinfo@sqe.com www.sqe.com
  • 2. Adam Richards Critical Logic Adam Richards is the director of consulting services at Critical Logic, a consulting firm specializing in model-based testing tools and services. With nearly two decades in the industry, Adam has experience as a developer, business analyst, requirements manager, test analyst, and QA manager. He brings first-hand experience implementing model-based testing approaches on systems in a wide array of industries, across the entire gamut of development methodologies.
  • 3. 9/20/2013 Model-Based Testing Concepts, Tools, and Techniques Presented by Adam Richards Starwest, 2013 2013 Critical Logic If the shoe fits Model-Based Testing offers something for everyone. 2013 Critical Logic 2 1
  • 4. 9/20/2013 Survey says Who uses MBT, how, and for what 2013 Critical Logic 3 Domains Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012 http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf 2013 Critical Logic 4 2
  • 5. 9/20/2013 Interfaces Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012 http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf 2013 Critical Logic 5 Methodologies Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012 http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf 2013 Critical Logic 6 3
  • 6. 9/20/2013 Test Level At what scope do you use MBT? Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012 http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf 2013 Critical Logic 7 Test Aspects Which aspects of the SUT do you test with MBT? Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012 http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf 2013 Critical Logic 8 4
  • 7. 9/20/2013 Benefits On average, respondents report MBT: Reduced escaped bugs by 59% Reduced testing costs by 17% Reduced testing duration by 25% Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012 http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf 2013 Critical Logic 9 Okay But what is it??? 2013 Critical Logic 10 5
  • 8. 9/20/2013 Model Soup What we Mean by Model 2013 Critical Logic 11 Structural Models 2013 Critical Logic Static ERD, Class, Component, Data Model Can define test environment (e.g. UTP) Foundation for Behavioral models Used for both Rqmts & System Models 12 6
  • 9. 9/20/2013 Behavioral Models Describe flow BPM, Activity, State, Cause-Effect Can specify abstract test scenarios (no data, test environment) Used for both Rqmts & System Models 2013 Critical Logic 13 Test Models 2013 Critical Logic Behavioral model + inputs + environment Specify tests at executable level Automatable May or may not include test generator 14 7
  • 10. 9/20/2013 MBT Process Menu* Hanging the model types together *Substitutions welcome 2013 Critical Logic 15 Flavor 1: Model-Centric Rqmts Model Rqmts Model Rare today (yet) Generate all Big investment Big returns 2013 Critical Logic Maint. costs Human error Defect escapes Defect introduction 16 8
  • 11. 9/20/2013 Flavor 2a: Model-Driven Less uncommon No rqmts model Generate executable tests from test model 2013 Critical Logic 17 Flavor 2b: Model-Driven (Agile) Test model only Generate acceptance criteria Maintain model, not tests Supports CIT 2013 Critical Logic 18 9
  • 12. 9/20/2013 Flavor 3: Model-Based No test model Walk rqmts model Validation later Add data, env. 2013 Critical Logic 19 MBT Flavor Rainbow Model Centric Model-generated tests & code Model Driven Model-generated tests Model Based Tests based on rqmts models, built by hand 2013 Critical Logic 20 10
  • 13. 9/20/2013 Tool Ingredients What Test Modeling Tools Can Do 2013 Critical Logic 21 Test Model Tool Functions Test Data Generators Test Case Editors Test Case Generators >> Test Coverage Criteria 2013 Critical Logic 22 11
  • 14. 9/20/2013 Test Coverage Criteria Sys/Rqmt Model Coverage Control Flow, Data Flow Transition-Based (FSM, statecharts) UML-Based (AD, Class, Seq, etc.) Data Coverage Boundary Value Analysis Classification Tree Statistical, Pairwise Fault-Based Coverage Mutation, Stuck-At Fault Model, MC/DC Requirements-Based Coverage All Requirements Coverage Utting, M.; Legeard, B.: "Practical Model-Based Testing - A Tools Approach". Morgan Kaufmann Publ, Amsterdam, 2007 2013 Critical Logic 23 Test Modeling Tool Examples Commercial RBT/DTT (Bender RBT, Critical Logic) Rhapsody ATG (IBM) Others (Conformiq, AETG, Leirios) Open Source ParTeG (Eclipse Plugin) Spec Explorer (Visual Studio) Others (JUMBL, Torx, Uppaal TRON) 2013 Critical Logic 24 12
  • 15. 9/20/2013 More Survey What kind of MBT tools were used? 17% Open Source 30% Custom 53% Commercial Testing modes used on project (avg)? 33.6% MBT 36.7% Traditional 29.7% Manual Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012 http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf 2013 Critical Logic 25 Coverage What kind of coverage is measured for your MBT test runs? Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012 http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf 2013 Critical Logic 26 13
  • 16. 9/20/2013 More Survey (last one) How do you connect MBT output, SUT control interfaces, & SUT observation interfaces? 06% Manual input & observation 15% Manual scripting required 15% Tool generates code/data + manual processing 26% Tool generates code/data for the SUT interface 38% Tool generates code/data for an adapter that abstracts the SUT interface Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012 http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf 2013 Critical Logic 27 Tool Challenges Opportunities Tool Integration Automation, RMS, TMS, SDLC, TestIF Coverage limited to source model Can it synthesize sources to generate tests? Constraints & infeasibles Good coverage executable tests Environment Independence What if the scripting language changes? 2013 Critical Logic 28 14
  • 17. 9/20/2013 Test Model Model-Centric Example - CEM 1. Multi-Source Synthesis 2. RMS Traceability 3. Manual & Auto 4. TMS integration 5. MCDC tests that run 6. Language independence 2013 Critical Logic 29 Yeah but Advantages, Challenges, & Benefits 2013 Critical Logic 30 15
  • 18. 9/20/2013 MBT Advantages 1. Flexibility (env, interface, model basis) 2. Low barrier to entry (80-100 hrs) [Binder] 3. Measurable, repeatable test coverage 4. Basis for effort estimation 5. Review models, not tests 6. Move validation forward in SDLC 7. Store test definitions as data 8. Integrated test automation 9. Maintain models, not tests 2013 Critical Logic 31 MBT Challenges Education of test modelers MBT strategy is an intellectual task Expectations: models not automatic Selecting and integrating tool suite Integration with existing SDLC 2013 Critical Logic People Tools Processes 32 16
  • 19. 9/20/2013 MBT Benefits 1. 2. 3. 4. 5. 6. 7. 8. 9. Requirements improvement Efficient test generation, automation Reduce maintenance costs Models as basis for effort estimation Reduce technical debt Lower cost of quality from rework Reduce defect escapes Reduce defect introduction Data-based result analysis and reporting 2013 Critical Logic 33 Tips to Get Started Start small Pilot low-risk app/change Rev. eng. an old complex function Parallel = ideal Use a vendor Planning, objectives, strategy Delivery, training & mentoring Identify demonstration objectives 2013 Critical Logic Test Coverage improvement Requirements improvement Reduced defect introduction/escape rates Scalability, strategy 34 17
  • 20. 9/20/2013 Discussion 2013 Critical Logic 35 Thank You! Adam Richards Critical Logic Tel: 415-814-9556 arichards@critical-logic.com 2013 Critical Logic 36 18
  • 21. 9/20/2013 References & Further Reading Ambler, Scott W. Agile Modeling: Effective Practices for Extreme Programming and the Unified Process. 2002 Utting, M.; Legeard, B.: "Practical Model-Based Testing - A Tools Approach". Morgan Kaufmann Publ, Amsterdam, 2007 Web site of Robert V. Binder, software engineer, expert on testing and automation, source of important research information in this presentation: http://www.robertvbinder.com/ http://www.robertvbinder.com/open-source-tools-for-model-based-testing/ http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf Web site of Stephan Weissleder, MBT theoretician, expert, and innovator: http://model-based-testing.de/ Please feel free to contact me directly if you have any questions or simply to further the discussion. ModelBased Testing is the future! 2013 Critical Logic 37 19