Slide 1

Michael Biwer Software Quality Assurance Automated Testing Techniques Slide 2 Apology I did not communicate to Dr. Yang my Abstract and References by Monday 8:00AM. This is because I completely forgot and dropped the ball on the simplest part of this course. This is why there had been a late communication saying that I was to present today. Thankfully, Dr. Yang and me worked out a deal. Slide 3 Agenda Software Quality Assurance: Automated Testing Overview Testing Methodology Overview of types of tests Automated Testing Software Versus Manual Testing, Pros, Cons, And Basic Functionality Types of Automated Testing Framework Modular, Data-Driven and Keyword-Driven Frameworks. Slide 4 Definition The management and performance of test activities, to include the development and execution of test scripts so as to verify test requirements, using an automated test tool[2]. Slide 5 History Began to evolve with the rest of software development as pressure was put on developers to: Reduce time, cut costs and increase quality. This began to create more incremental builds: Shown with the increase of Rapid Application Development methods such as Agile. Posed the difficulties of faster changes and additions to project requirements. Higher customer appreciation of quality control. (ISO 9000) Therefore, Software Quality Assurance: Automation Testing began to evolve and third party companies began to create automation suites. Slide 6 Developer Testing VS. Quality Assurance Testing Developer Testing: Ensures that the all of the projects requirements have been met through testing the processes. Usually, done through testing the code from an internal standpoint. Overall, looks to ensure that the process work correctly. Quality Assurance Testing: Ensures that all of the project requirements have been met through testing the product. Usually, done through testing the product from an external standpoint. Most instances mimics users view of the product. Overall, looks to improve the product as a whole. Slide 7 Manual Testing Vs. Automated Testing Manual: Labor intensive, testing take many man-hours to complete. Does not require software knowledge or skills from the testers. Repetitive tests still require many hours to complete and may be narrowed to fit a time schedule. Automated: Capital intensive. Large upfront cost for the software. Scalability might require hardware upgrades. Will not receive any benefits until the 2 3 project, because the creation of the test framework. Repetitive testing shows rates of returns around the 10th time it is run. Requires more specialist knowledge bases for the automation engineer. Slide 8 Automated TestingManual Testing Test Case Creation 1Need for test created 2Business Analyst creates test requirements, test plan and test cases which reflects the need in automated testing tool software. Business Analyst creates test requirements, test plan and test cases which reflects the need in a document. 3Automation Engineer creates test script framework which reflect the test plan and test cases requirements. Manual Tester carries out the test plan, recording results of tests. 4Automation Testers create driver scripts to run the test scripts generated by the automation Engineer. Results of tests and expected results are compared and bugs are recorded for the next build. 5 Tests are run automatically and results are analyzed. Business Analyst logs bugs for next build/run of test. N/A All Runs of the Test Case Afterwards 6 The Business Analyst reviews test, makes adjustments if needed and re-run the test in the automation software. The Business Analyst reviews test, makes adjustments. 7 N/AManual Tester carries out the test plan, recording results of tests. Slide 9 Why? Proper defect tracking. Aim to enhance the overall quality of the product, not to just ensure it works correctly. Different eyes on the testing ensures that project requirements were communicated efficiently between customer and development team. Because not all developers are good testers, it is good to have a specialist test the product to ensure consistent quality. Ensures that the customer has input, in order to make sure that they will be happy with the end result. Slide 10 Pros 24/7 Testing. Repetitive testing is easily accomplished and cost effective in long running projects or in maintenance. In depth reporting tools. Consistent testing. Fewer, more skilled workers. Slide 11 Cons Cost Heavy upfront cost with the purchase of the software and the time devoted to the creation of a automation framework. In addition, you will not see a return on investment right away. Must pay your employees more. You will require an automation engineer which has software programming knowledge. You will have to train your automation testing team in the automation software tool you choose. Slide 12 Overview of Testing Types Two overall types of testing: White-Box Testing: Testing that is done from an internal viewpoint of the application. Black-Box Testing: Testing that is done from an external viewpoint of the application with only the working knowledge of how the application should work. InputOutput InputProcessOutput Slide 13 Overview of Testing Methods (cont..) Unit Testing Tests that target specific code to ensure that it can operate separately from the system given the correct inputs. Integration Testing Testing that aims to test the applications code as a whole to ensure that the components interact correctly. System Testing Testing that makes sure that the application meets the requirements set down for the software and that it doesnt interfere with any other platforms/applications. Acceptance Testing Testing where either the user or developer/QA team is given a build of the program in order to verify the requirements have been met. Slide 14 Overview of Testing Methods(cont) Smoke Testing: Build Verification Testing, Ensures that no serious problem has occurred between build of the application. Regression Testing Testing normally performed after a major code overhaul has been made to make sure that there are no persisting defects and that no functionality has been lost. Functional Testing Testing that only cover a particular flow or process within the program. Non-Functional Testing Testing that confirms the overall quality of the product in reference to the user. Performance Testing Testing that checks the scalability, reliability and resource usage of certain aspect of the program. Usability Testing Testing of the Graphical User Interface (GUI) works correctly in the given layout. Slide 15 When to Automate a Test It makes the most sense to automate tests for products that are going to have continuous development or maintenance. This ensures that the test cases can be reused, increasing the return on investment of the automation software. Such as for the purposes of regression or smoke tests. Side Note: Developers often use automated testing methods for unit or integration tests. (Junit Tests.) Slide 16 Basic Automation Software Features Record/Playback Process: Click the play button in the automation suite. The Automation Software tracks your mouse movements, clicks and keyboard actions. Click the stop button when the test completes. The Software then compiles the movements into a test cases for you to insert into a test plan. You are then able to perform those exact actions multiple times. Sometimes features of the automation suite include auto-generation of record/playback tests into test scripts. Slide 17 Basic Automation Software Features Record/Playback (cont) Benefits: Simple, does not require a software background to utilize. Quick, there is not much set up to create this type of test. Can be used for repetitive testing on a basic level. Drawbacks The types of tests that you can performs are limited to external options. If you have to change the test data, you must redo the entire test. Therefore, there can be a lot of maintenance. Slide 18 Example of Record/Playback http://youtu.be/facNRr8SHpk?t=45s Slide 19 Basic Automation Software Features Object Recognition Allows you to have an external look at the objects that are currently being displayed in the application being tested. Aka. GUI. You are able to store these objects in an Object Repository in order to easily reference them or you are able to Call them directly within your test scripts. They are formed in a hierarchy of container objects. Automation software with this ability gives you libraries which allow you to perform functions and retrieve/change properties of the objects. Slide 20 Basic Automation Software Features Object Recognition(cont) Benefits: Allows you to set up more in-depth specific tests within test scripts. Drawbacks: Requires a test script framework to create test on these objects. Creates large initial overhead. Problems occur when custom objects/properties are created by the developers which the automation suite cannot recognize. This would require you to create your own Object Script Extender to recognize the object/property. Slide 21 Example of HP QuickTest Pro Object Recognition Slide 22 Basic Automation Software Features Test Script Execution Framework Allows you to create libraries in order to test any portion of your application. Code GUI APIs Only limited to the amount of coding you want to do. These scripts are accessed through drivers and can be used across many different test cases. Commonly these test scripts are created using VBScript, Java, Perl, or CGI. The automation tools have pre-loaded libraries to give you extra capabilities with testing objects. Slide 23 Simple test script example. Uses a direct calling of objects to interact with them. Slide 24 Basic Automation Software Features Test Cases Reports Able to view the passes and the fails of your automation tests in a report format. Since this is based on your test scripts, you only get as much detailed information as you allow the test case record to collect from the tests. Slide 25 HP QTP: Example Report Slide 26 Components Test Plan A set of test cases which verify similar requirements of the application. Test Case A test to verify requirements of the application. Driver Scripts: The overall scripts that call your test scripts that make up your test cases. In essence, will reflect your test plan. Test Scripts: Code which allows for the tests actions to be carried out. Test Data: The data that you will be testing in your application. Slide 27 Meta-Language Development Meta-Language is a abstract language used in running scripts. It has been simplified for the average user with no knowledge of programing to use. This allows testers that only have the business knowledge of the system to utilize testing tools. Allows for manual testers to migrate onto the new system easily. This is essential for the testers to take ownership of the test cases. It is used at the highest visible level for the users to see, the Driver Script. In essence however, the Meta-Language, will be a list of parameters that will be parsed/translated and used in the test scripts. Slide 28 HP QTP Example: Input parameters that the automation engineer has set for this Driver Script Where the automation tester inputs the test parameters. Slide 29 Types: Modular Framework Creating specific test scripts in a library which are used within the Driver Scripts to perform the tests. IE: In the windows calculator, having a script that would verify that 5 + 5 = 10. Benefits: Quick and easy to put together. Drawbacks: Difficult to maintain. Less reusable code. Data is within the test scripts. Can easily have multiple copies of the same code. Because the test scripts have all of the control, this means that the ownership of the test cases lie in the automation engineer. Slide 30 Types:Modular Framework (cou) Driver Script Test Library 1Test Library 2 Function_1Function_2Function_3Function_4Function_5Function_6 Example of framework setup: Example of Psuedocode for a Modular Framework Driver Script: Main Read Parameters Window_1 Functions Function_1 Call Function_2 Call Function_3 Call Window_2 Functions Function_4 Call Function_5 Call Function_6 Call End Main Slide 31 Action_TypeName_Of_Script Run_ScriptVerify_Calc_Script Example of Driver Script Parameters: Slide 32 Types: Data-Driven Framework This framework looks to fix the problem of data being within the test script. The input is commonly moved to the driver script or to an external database or spreadsheet which is fed into the test scripts. IE. In the windows calculator, you have a spreadsheet with numbers which are run through a script that adds them together. Benefits: Gives ownership of the test cases to the testers. Maintenance is more manageable. Drawbacks: Test case information can be split into different areas confusing large operations. Scalability is a real problem. Slide 33 Types: Data-Driven Framework (cou) Driver File Script_1Script_2Script_3 Data_File_1Data_File_2Data_File_3 Comparison of Expected and Actual Outputs Example of framework structure: Pseudocode for Data-Driven Driver Script: Main Read in Parameters Perform required Script with given parameters Script_1 Call Script_2 Call Script_3 Call End Main Slide 34 Script_NameData_FileNumbersExpected_Result Add_Calc_ScriptValue_Add_To_10.xmlA1-A2B1 Sub_Calc_ScriptValue_Sub_To_0.xmlA1-A2B1 Mul_Calc_ScriptValue_Mul_To_49.xmlA1-A2B1 Example of the Data-Driven Driver Script: Slide 35 Types: Keyword-Driven Frameworks This framework looks to provide maximum ownership to the testers and maximum flexibility. Does this by providing a object by object look at the test cases and providing actions that can be taken out on those objects. Benefits: Provides all ownership to the testers. Maximum flexibility and highly scalable. Maintenance is on the testers and is standardized. Drawbacks: Large overhead time for creating the framework. A driver file must be created for each window of the application. An iterative approach which can increase test case execution times. Slide 36 Types: Keyword-Driven Framework (cou) Pseudocode for Keyword-Driven Driver Script: Main Read in Parameters Parse parameters into correct Object+Function+Data groups For Each Object in window Selected Action Object_1 Action Calls Object_2 Action Calls Object_3 Action Calls End Main Slide 37 Iteration Action_TypeButton_1Edit_1CheckBox_1 1 Click 2 Insert Hello 3 Click Highlight 4 Key_CNTRL_X 5 click Click 6 Key_CNTRL_V 7 Property Text=HelloChecked=True Slide 38 Types: Hybrid Framework Combines the previous frameworks together in various levels to provide maximum testing capabilities. Normally involves connecting to a test database for various inputs. Example of Hybrid Framework Psuedocode: Main Read Parameters Group Parameters Link to database Select a specific window For each object in a specific window Perform the needed scripts End Main Slide 39 What we discussed Software Quality Assurance: Automated Testing Overview Testing Methodology Overview of types of tests Automated Testing Software Versus Manual Testing, Pros, Cons, And Functionality Types of Automated Testing Framework Slide 40 References 1. Chaos manifesto 2012: The year of the executive sponsor. (2012). Retrieved from http://versionone.com/assets/img/files/CHAOSManifesto2012.pdf 2. Elfriede, D., Rashka, J., & Paul, J. (2008). Automated software testing: Introduction, management and performance. (13 ed.). Castleton, New York: Hamilton. Retrieved from http://books.google.com/books?id=kl2H0G6EFf0C&lpg=PR5&ots=wN_PrWrBx1&dq=automation software test qa&lr&pg=PA43 3. Mosley, D., & Posey, B. (2002). Just enough software test automation. Upper Saddle Hill, new Jersey: Prentice Hall PTR. Retrieved from http://books.google.com/books?id=PEBvfWESIt4C&lpg=PR15&ots=xGxZPGvp2P&dq=software test automation &lr&pg=PA3 4. Laukkanen, P. (2006). Data-driven and keyword-driven test automation frameworks. (Master's thesis)Retrieved from http://eliga.fi/Thesis-Pekka-Laukkanen.pdf http://eliga.fi/Thesis-Pekka-Laukkanen.pdf 5. Software testing. (n.d.). Retrieved from http://en.wikipedia.org/wiki/Software_testinghttp://en.wikipedia.org/wiki/Software_testing 6. Allott, S. (1999, 10). Automate your tests - you wont regress it!. Seventeenth annual pacific northwest software quality conference, Oregon Convention Center Portland, Oregon. Retrieved from http://www.snapshots.pnsqc.org/2011-theme/pnsqc-04-06- 2011/proceedings/pnsqc99.pdfhttp://www.snapshots.pnsqc.org/2011-theme/pnsqc-04-06- 2011/proceedings/pnsqc99.pdf 7. Kumar, S. (Designer). (2007, April 9). File:SampleCode.JPG [Print Photo]. Retrieved from http://en.wikipedia.org/wiki/File:SampleCode.JPG http://en.wikipedia.org/wiki/File:SampleCode.JPG 8. Kiranpaul, K. (n.d.). Quick test professional tutorials - qtp fundamentals, qtp features, vb scripting examples in qtp, qtp framework and vb script.. Retrieved from http://qtp4free.blogspot.com/2010/05/object-spy-in-qtp-92.htmlhttp://qtp4free.blogspot.com/2010/05/object-spy-in-qtp-92.html 9. Jain, A. (Photographer). (2013, July 5). Good Looking and Enhanced Results Viewer [Web Graphic]. Retrieved from http://2.bp.blogspot.com/_iTz5d0BSy6g/TPY5-7SOIGI/AAAAAAAAAPM/wxlEl9BHMqM/s1600/QTP_11_RV_SS.png http://2.bp.blogspot.com/_iTz5d0BSy6g/TPY5-7SOIGI/AAAAAAAAAPM/wxlEl9BHMqM/s1600/QTP_11_RV_SS.png 10. Walter, T. (Photographer). (2012, July 6). Add Parameters to specify the Ranorex Automation [Web Graphic]. Retrieved from http://www.ranorex.com/blog/wp-content/uploads/2012/07/Add-Parameters-To-Test.png http://www.ranorex.com/blog/wp-content/uploads/2012/07/Add-Parameters-To-Test.png 11. Tutor. (2012, Aug 6). Software testing life cycle (stlc) complete tutorial. Retrieved from http://www.sdlc.ws/software-testing-life- cycle-stlc-complete-tutorial/