Upload
mitchell-waters
View
220
Download
0
Tags:
Embed Size (px)
Citation preview
Sakai CommunityPerformance Testing
Working Proof of Concept
Sakai Performance Working GroupLinda Place, Chris Kretler, Alan Berg
Universities of Michigan & Amsterdam
1 July 2008
Performance Testing Overview
• Part of QA process– May include stress, capacity, scalability, reliability
as well as performance (response time, throughput, bottleneck identification)
• Black box testing– Running system with projected use-case
scenarios for acceptable throughput and response times• How will users experience application?
Performance Testing Overview
• White box testing– Pushing system to identify application, database,
operating system, or network problems• Tune environment to identify and address specific
problems• Tests watched by developers, DBA, system and network
administrators, and performance engineers
• Resource intensive nature of process– Infrastructure, personnel, tools, time
Community Performance Testing
• The Approach
• Common Environment/Data Creation
• Load Test Tool
• Proof of Concept
• The Practice
Organization vs. Community Testing
• Organization approach– Advantages
• Focus on exact infrastructure used in production• Define use cases according to real use in production• Only test tools specific to organization
– Limitations• Very resource intensive (expensive)• Hard to maintain• Hard to be agile without careful advance planning
5
Organization vs. Community Testing
• Community approach– Advantages
• Pool of test scripts available for immediate use• May not need full testing infrastructure to be confident in
production• Increased confidence in adding new tools• Total cost of testing shared
– Limitations• Must clearly communicate results to community• Collaboration seems harder than just doing it yourself
6
WG Project Objectives
• Create QA environment that enables performance testing by community in shared manner– Plan available on Confluence
• http://confluence.sakaiproject.org/confluence/display/PERF/Home
• Have working proof of concept by Paris Sakai Conference
7
Infrastructure Issues
• What do we learn from testing done on infrastructure that doesn’t match our own?– Advantages
• Software/application focus forces code improvements• More efficient code more likely to scale to meet size and
performance needs
– Disadvantages• Hardware and network bottlenecks NOT found• Capacity limits of environment NOT discovered
Community Performance Testing
• The Approach
• Common Environment/Data
Creation
• Load Test Tool
• Proof of Concept
• The Practice
Provisioning
• Alan's provisioning tool– Perl scripts use web services to populate DB
with courses
– Configuration via property files
– Adds tools, instructors and students to courses
– Creates file structure and uploads resources to
sites
• Other efforts are underway
Provisioning, Benefits
• Easy, fast creation of test environment
• Common data environment
– Common environment by sharing properties
files
– Common data files for tests
Provisioning, Some Modifications
ORIGINAL
1. provision.pl
2. make_media_list.pl
3. make_local_resource
s.pl
MODIFIED
1. provision.pl
2. make_media_list.pl
3. create_templates.pl
4. load_resources.pl
Data Environment Design
• Based on UM Winter 2008 semester– Number of courses (2000) and students (20,000)– Tool mix (+projection)
• Courses broken into 5x3 categories:– 5 categories of roster size– 3 categories of tool distribution
• Site name based upon category
Community Performance Testing
• The Approach
• Common Environment/Data Creation
• Load Test Tool
• Proof of Concept
• The Practice
Evaluation & Tool Selection
• Goal– Enable near-enterprise level of tool quality using
Open Source tools– Hewlett Packard LoadRunner used for comparison
15
Evaluated Test Software
Grinder
JMeter
CLIF
Web Application Load
Simulator
SlamD (Sun)
Funkload
Selenium
Pylot
WebLoad Open Source
TestMaker
OpenSTA
Hammerhead
httperf
Seagull
Deluge
Grinder: Plus
• Easy record mechanism
– Can record https on multiple platforms
• Like the scripting language (jython)
– Java extensible
– Allows for re-usable libraries
– Flexible reporting, data handling
• Run-time UI displays multiple test scripts
– We use 22 for our standard scenario!
Grinder: Plus
• Distributed load generation
– Multi-platform
• Active user/development community
– Open source
– Separate project for developing reports:
“Grinder Analyzer”
• Jython Eclipse plug-in (Grinderstone)
Grinder: Minus
• Default recorded script is complex
– Verbose results data
– Perl script cleans the default record script
• (J)Python not ubiquitous
– To do list shows support for java scripts
• Reporting project doesn't consolidate data
easily
Community Performance Testing
• The Approach
• Common Environment/Data Creation
• Load Test Tool
• Proof of Concept
• The Practice
Proof of Concept
• Read-only scripts used– Download file– Viewing a grade– “Idle” user
• Mix/Frequency of users corresponds to a peak usage period– Events table used to determine usage
Event Graphs
LoadRunner Script
Grinder Script
LoadRunner Results
Grinder Results
Database Results
Grinder: Some Ideas
• Consolidated & Expanded reporting
• Business Process Testing
– Makes test script development more flexible
– Potentially expand the pool of testers
Community Performance Testing
• The Approach
• Common Environment/Data Creation
• Load Test Tool
• Proof of Concept
• The Practice
Establishing Baseline(s)
• Identify “core” scripts for Sakai baseline• Identify related script clusters to
baseline tool clusters
• Maximum time threshold for page loads
• Minimum user load for tools and kernel
• Database activity reports
Establishing Baseline(s)
• Identify “core” scripts for Sakai baseline– All tools included in release?– More limited subset?
• Identify related script clusters to baseline tool clusters– Tools with dependencies?– Tools with similar actions?– Usage scenarios associated with specific
campuses?
32
Community Sharing
• Scripts– Set up performance test branch in Sakai svn?– Associate test scripts with code in svn?– Expand Confluence Performance WG site?– Establish communication mechanisms during QA
release cycles?
33
Community Sharing
• Scripts– Sakai performance test branch in svn?– Packaged along with tools in svn?
• Results– Expand Confluence site?– Contribute to QA “score card”
• Documentation– On confluence and/or in svn?
Contributing to Sakai QA
• Recommend performance “standards” to be part of formal QA release process?– Score cards
• Volunteer organization to test specific tools based on usage needs– Use shared scripts and contribute scripts/results
back to community
• Combined test results yield greater confidence in Sakai code
Moving from Concept to Community
• Performance Working Group BOF– Wednesday, 11-12, BOF Room 1
• Check BOF schedule for exact location
• Looking to identify contributors• Goal to establish new working plan for WG
contribution to next conference
Michigan’s Contribution
• Leadership & initial design• All test scripts used at Michigan• All results from Michigan tests• All documentation needed to conduct testing
as currently implemented at Michigan• All relevant materials for future tests• Kernel testing for Next Generation Sakai