View
5
Download
0
Category
Preview:
Citation preview
Disclaimer• THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
• WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.
• ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES.
• ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.
• IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.
• IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
• NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
• CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS
Development Transformation
• Drawn from my work in IBM Runtimes and on Eclipse OMR, Eclipse OpenJ9, and AdoptOpenJDK projects
• Hundreds of thousands of tests, on thousands of test machines representing 18+ platform variants, multiple versions of Java (v5 – 10)
• Goal to simplify, redesign and decouple from legacy/proprietary tools and processes and create open, agile, flexible tests and test solutions to empower developers
AdoptOpenJDK
What is a microservice?
• A succinct API to accomplish a particular task (happiness knows no bounds, but microservices do…)
• A set of microservices assembled together can create a whole system (replacing a monolithic approach)
• While often used in concert, each service can be used independently
• Benefits: Modular, Simpler, Portable, Customizable Apps, EasierDevelopment & Deployment, Co-operative
ResultAnalytics
Data
Services
UI Layer
Cores
raw refined
custom dashboardother clients
TestGeneration BenchEngineCoreAnalytics
TestSelection
BugPrediction
InputOptions
ResultSummary
ResultCompare
githubrepos
Jenkins servers
Microservices Architecture
Activities (and Questions) Related to Test
Plan Implement Automate Execute
Triage Exclude
Report
What? How?
How often? How easy?
How few? How fast? What failed? Why?
What next?
Decompose into a set of services by test activity,services to help answer the questions
Planning Phase
What tests do you need?
How are people using your product? What are their user stories?
What additional info do you have (code coverage, code churn information)?
Plan Implement Automate Execute
Triage Exclude
Report
What?
‘Planning Phase’ Microservice Examples
• Core Analytics Service – inspect java core files from service reports, gather information on usage, particularly patterns of input parameters
• Bug Prediction Service – based on research* an inexpensive algorithm to find hot spots (most changed files) in code base, applied to any given repo
*BugCache for Inspections: Hit or Miss?
Implementation Phase
How best to design and write your tests? What tools/frameworks to use?
How to minimize number of tests while maximizing functional coverage?
Plan Automate Execute
Triage Exclude
ReportImplement
How?
‘Implementation Phase’ Microservice Example
• Test Generation Service – assist developers in the onerous task of writing thorough functional tests, standardized test naming and structure, CTD for minimal tests with good functional coverage• Upload header files (function/method signatures)
• Inspect input types (primitive or complex)
• Generate combinations of inputs (pairwise/CTD)
• Generate partial test code… (with standardized naming, parameterized inputs)
Automation Phase
• How do I add my tests into a CI workflow?
• How do I replace old practices with new improved ones?
Plan Execute
Triage Exclude
ReportImplement Automate
How often? How easy?
‘Automation Phase’ Microservice Example
• BenchEngine Service – make it easy for anyone to run performance benchmarks • generate scripts to run benchmarks from definition files
• allow for variations and specific platform settings
• send scripts to Jenkins nodes to run automatically run benchmarks
BenchEngine
Developer: • fires up browser• connects to BenchEngine service• select inputs
Jenkins
input selections:• benchmark definition file• action type
generateScript:• local run • insert into automated
builds
submitJob
BenchEngine Service
Execution Phase• What is the minimal set of tests I need to run, given a particular change?
• What input options do I run with?
• What is the execution time? Has it changed over time?
Plan
Triage Exclude
ReportImplement Automate Execute
How few? How fast?
‘Execution Phase’ Microservice Example
• Test Selection Service – change based testing, uses code coverage information to generate a smaller/smarter set of tests to run on a pull request
• Input Options Service – grabs input options defined in tests at start of a build, names and stores them (unique sentences of options), for sharing with other builds/tests
TestSelection
Developer: • fires up browser• connects to Test Selection service• select inputs
Jenkins
input selections:• pull request in github• platform(s)
submitJob
Test Selection Service
• (nearly) headless service to feed information to builds• not all services have (or need) a pretty GUI
InputOptions
Jenkins mongodb
Input Options Service
Triage Phase• Has this failure happened before? Is this failure intermittent? If so, are there
recognizable patterns?
• Is there other information besides test output that I can use to help find patterns?
• Do I need to exclude this test?
• Can I re-include it automatically when the issue is fixed?
Plan Execute ReportImplement Automate
Triage Exclude
What failed? Why?
‘Triage Phase’ Microservice Example
• WIP: Results Analytics Service - automate the “googling of error messages”, look for patterns• search across builds at one server?
• search across builds at multiple servers?
• search the internet?
• Use Watson pattern matching / analytics engine...
• If you didn’t want all of the above, then Jenkins plugin?
Reporting Phase
• Can I summarize a large/diverse set of tests?
• Where can I see the test report?
• Can I compare test results easily? From many data sources?
Plan Execute
Triage Exclude
Implement Automate Report
What next?
‘Report Phase’ Microservice Example
• Results Summary Service – summarize the results of multiple builds running on multiple servers, overlay customizable dashboard
• Results Comparison Service – given more than one result of same test, compare them. Examples of results to compare:• FV tests from Build 1 compared with those of Build 12
• Scala (or other 3rd party app) community tests running on OpenJDK vs OpenJDKwith OpenJ9 (Java8 / Java9)
• Last 4 BenchmarkXYZ runs against OpenJ9 (trend)
• Last several months of GC perf stats from OMR
Results Summary Service
• Model the common elements (status, execution time, failure age, etc) for uniform representation
• Leverage Jenkins API
• Progression of enhancements• Raw data• Simple diff• Graphical representations• Overlay customizable dashboard, simple elements, continuous feedback
Results Comparison Service
Demo
• Gather perf metrics from Eclipse OMR builds
• Store historical data in mongodb• GC perf dashboard allows many
different views onto the data
Review: Examples of microservices
Plan Implement Automate Execute
Triage Exclude
Report
Core analytics serviceBug prediction service
Test Generation Service Test Selection ServiceInput-Options Service
Result Analytics Service Result Summary ServiceBenchEngine
Microservices Examples Review
ResultAnalytics
Data
Services
UI Layer
Cores
raw refined
custom dashboardother clients
TestGeneration BenchEngineCoreAnalytics
TestSelection
BugPrediction
InputOptions
ResultSummary
ResultCompare
githubrepos
Jenkins servers
Plan Execute Triage ReportImplement Automate
Questions Addressed
• What aspects of your testing can be shared across multiple products? Decompose by common activities…
• How to identify, abstract, and convert functionality used by many test systems into shared services? Common patterns (apps take inputs, tests produce results)...
• Why bother with microservices? In a cloud? Modular (super-fast dev), portable, assemble for a ‘system’ or singular use, leverage other services…
• Gotchas... Network reliance, fallbacks needed…
Recommended