Testing Virtual Reality - Software Testing Virtual Reality (And other complex software environments!)

  • View
    0

  • Download
    0

Embed Size (px)

Text of Testing Virtual Reality - Software Testing Virtual Reality (And other complex software...

  • CONTACT Emil Alégroth – Emil.Alegroth@Auqtus.com

    WEB www.auqtus.com

    Testing Virtual Reality (And other complex software environments!)

    Dr. Emil Alégroth Auqtus | Blekinge Institute of Technology | Chalmers University of Technology

  • • We delivering state-of-art testing solutions to industry with focus on Visual GUI Testing (VGT).

    • Founded by academics and technical experts in the area of automated testing. – Combined; over 50 years of academic experience – 100s of academic publications – World-leading researchers on VGT (Founded the term)

    Contributors to the talk:

    Dr. Emil Alégroth Prof. Robert Feldt Dr. Simon Poulding

  • What is Virtual Reality?

    • Oculus rift and Google Cardboard • Leisure applications

    • Games • Communication

    • Learning applications

  • Our application: VRHangout

  • Unity

    • Game engine

    • All in-game fixtures treated as objects – Makes scripting easy

    • C# scripting – Quick and powerful

    – Same syntax as java

    • Many plug-ins – Google cardboard!

    • Main limitation – Standard components are generally slow!

    Unity game: Hearthstone

  • My experiences with Unity (5h vs 5 days)

  • Unity

    VR is only the beginning!

  • Augmented reality

    • Leisure applications (e.g. Minecraft) • Life-critical applications

    • Medical systems

    Safety requirements will require stringent quality assurance!

  • How do we test in this environment?

    • Low-level: – Automated unit tests and integration tests – C# components (in Unity)

    • High-level: – Manual testing 

    • Why? – The real-time, virtual environment, is (often) non-

    deterministic and complex to test – Complexity: Computationally heavy to test, difficult to find

    oracles, etc.

  • A complex environment requires complex testing

    • Search-based software testing (SBST) – Test case generation, based on a (*)fitness function, that

    can automatically determine if the output of the generated test case is (**)correct/faulty/interesting to/for the tester.

    • What is a (*)? – Domain model that defines rules for how to, for instance,

    generate suitable test cases (A heuristic). Can be modified to learn during execution (Machine learning).

    • How do we determine (**) in a VR environment? – For instance: Kolmogorov complexity!

  • G

    Test case generation with Meta- Heuristics

    • Meta-heuristic – A description of a possible solution to generate test cases – For instance: Genetic algorithms

    T1 T2

    T2 T1 T2

    T1

    T2

    T1 T2T1 Determine best child with a fitness function (best fit)

    Gen 1 seed

    Gen 1

    Gen 2 seed T2T1T1

  • Oracle?

    • Oracle – A means of determining test case outcome – Examples: Value, image, strings or other output

    • Kolmogorov complexity allows you to determine a distance between any two objects!

    • The measure gives a distance between two objects – Determine if a state is potentially correct/faulty/interesting

  • Kolmogorov complexity example usage

  • SBST + Kolmogorov

    • Generate exploratory test cases, verify with the automated oracle and learn by testing!

    • Small problem…

    – Kolmogorov komplexity can NOT be computed!

    – Theoretical concept...

    • …BUT! It can be approximated with compression!

    Compress( (1,[1,3,4,5]) , (2,[1,4,5])

    ) = ([1,4,5],(1,[3]))

    Compress( , )

    = ( ,(255,0,0),(0,0,255)) ”Distance”

    ”Distance”

    No tools do this yet but they are on the way!

  • VR and AR are tomorrow. We need tools/techniques today!

    GUI

    GUI model

    System architecture

    System components /Source code

    System model

    Multiple languages

    Missing interfaces

    Too much code

    Etc.

    Encapsulated design

    Distributed system

    Etc.

    Custom components

    Limited usability

    Etc. Maintenance

    costs

    Speed Robustness Etc.

    FAILURES

    DEFECTS

    CHALLENGES

    MORE SUPPORT

    STRONGER NEED

    (e.g. Continuous

    development)

  • Visual GUI Testing (VGT): JAutomate

    • Seamless testing of different applications • End-to-end testing • Advanced testing through the GUI • Non-functional testing (e.g. Performance) • Etc.

  • VGT: Primarily System- and Acceptance Regression testing

    • VGT-GUITAR – Automated Exploratory, model-based, GUI testing

    – State space explosion problems – Conceptualization and Evaluation of Component-based Testing Unified with Visual GUI Testing:

    an Empirical study

    • What if we could provide the tester with a heads- up display to improve exploratory testing based on user domain knowledge?

    • What if we could automatically record exploratory test cases for regression testing?

  • Augmented exploratory testing

    Circles: covered states Color: Likelyhood of finding defect

    Expand: Get more information

    Fantastic!!! but is it a

    Fantasy???

  • Current state

  • Neural networks combined with advanced search algorithms

    Home

    v

    Neural net for Version X+1

    • Each node has: • A screenshot of the state of the SUT • The interaction to move to the next node

    COMING SOON!

    If advanced model, automatic exploration!

  • Conclusions

    • VR and AR require more complex test techniques

    • Search-based testing with Kolmogorov complexity

    – Not available yet!

    • Visual GUI Testing

    • Augmented Exploratory Testing

    Emil.Alegroth@Auqtus.com

  • Thank you! Questions?

    E. Börjesson, R. Feldt, “Automated System testing using Visual GUI Testing Tools: A Comperative Study in Industry”, International Conference on Software Testing, 2012. E.Alégroth, R.Feldt, H. Olsson, “Transitioning Manual System Test Suites to Automated Testing: An Industrial Case Study”, International Conference on Software Testing, 2013. E. Alégroth, M. Nass, H. Olsson, “JAutomate: a Tool for System- and Acceptance- test Automation”, International Conference on Software Testing, 2013. E. Alégroth, R. Feldt, L. Ryrholm, “Visual GUI Testing in Practice: Challenges, problems and limitations”, Empirical Software Engineering Journal, 2014 E. Alégroth, Z. Gao, R. A.P. Oliveira, A. Memon, “Conceptualization and Evaluation of Component-based Testing Unified with Visual GUI Testing: An Empirical Study”, International Conference on Software Testing, 2015. E. Alégroth, R. Feldt, P. Kolström, “Maintenance of Automated Testing in Industry: An Empirical study on Visual GUI Testing”, Journal of Systems and Software (In submission). E. Alégroth, J. Gustafsson, H. Ivarsson, R. Feldt, “Replicating Rare Software Failures with Visual GUI Testing: An Industrial Success Story”, IEEE-Software.

    Emil.Alegroth@Auqtus.com