Embed Size (px)
DESCRIPTION
The University of Toronto wanted a faster Blackboard Learn environment and needed to squeeze the most out of its hardware. Apache 1.3 that Blackboard bundles with its BB Learn was not up to the job. By upgrading to Apache 2.2, a new world of options become available. Now, Shibboleth 2 as a Single Sign-on, LDAP as a native authentication scheme for Mobile Learn and OpenSSL PKCS#11 Cryptographic support are all realities for Blackboard.The PKCS#11 support was particularly important, as is leverages the Solaris Cryptographic Framework (SCF) and transparently offloads cryptographic operations to available hardware providers included in the new Oracle T4 chip. UofT migrated all of its Blackboard VMs to a single Oracle SPARC T4-4 after upgrading to BB9.1SP8. The results were remarkable. A single box, 5 rack-U in height, delivers Blackboard to 80k users. Response times are down by 60% and we have no need for SSL-offload at the load-balancer.
Citation preview
Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU
Jose Manuel Lopez Lujan Senior LMS Coordinator���University of Toronto
Contact information
José Manuel López Luján • Email: [email protected]
• Twitter: @jmanuel_ll
• Blog: http://jose-manuel.me
• G+: http://gplus.to/josemanuel
2
John Calvin • Manager, Data Centres
• Email: [email protected]
University of Toronto
3
79,085 Students 3,229 Academic 5,224 Non-Academic 3 Campuses 7 Colleges
205 Undergraduate programs 79 Graduate programs
4,241,247 Sessions per month
1.5 M Hits/hr peak
2,153,536 Unique Visitors
13,313,110 Visits (Dec 11 Jun 12)
5.0 TB/hr peak
Where are we?
4
Canada 98.0100%
United States 0.7100%
China 0.23%
(not set) 0.1200%
Hong Kong 0.0900%
United Kingdom 0.0700%
South Korea 0.0600%
United Arab Emirates 0.0600%
University of Toronto Statistics
Chrome
Firefox
Internet Explorer
Safari
Android Browser
Opera
IE with Chrome Frame
Mozilla CompaNble Agent
Opera Mini
RockMelt
dem
ogra
phic
s technology
5
App1!
App2!
App3!
Data1!
Collab1!
F5!Load Balancer!
Hitachi 9985!
Infrastructure: Hardware Bb 9.1SP5
Infrastructure: Hardware
6
App1!
App2!
App3!
Hitachi 9985!2 Pools 2 RA!
300G FC !15k RPM!
!
Collab1!
Data1!
SPARC T3-4!4 CPU @ 1.65GHz!16 cores/CPU!8 threads/core!512 threads!512GB of RAM!
LDOMs!32 VCPUs!
64G of RAM!
LDOMs!80 VCPUs!
64G of RAM!
Bb 9.1SP5
7
Infrastructure: Software Bb 9.1SP5
Blackboard Learn 9.1 SP5
Apache 1.3
Pubcookie (DSO)
SSL
Apache 1.3 • No Compression with SSL • No Blackboard Mobile
PubCookie • SSO Solution • Hard to maintain • Custom Authentication Module
8
Looking for service and performance improvements
9
Apache 2.2.x • SSL and Compression working together
Shibboleth • Custom AuthenNcaNon Module for Bb
Bb Mobile
• Possible with Apache 1.3 and PubCookie? • Possible with Apache 2.2.x and Shibboleth?
MinificaNon • Worthwhile without compression?
The Plan
1 out of 4:
10
Using PubCookie Simple to administer
Force Web AuthenNcaNon SSO Page not mobile capable
PROS
CONS
Blackboard Mobile
<Location /webapps/Bb-mobile-bb_bb60>!!satisfy any!!AuthType none!!order deny,allow!!allow from all!
</Location>!
11
Enterprise!LDAP Server!
App4!
mobile.lms.utoronto.ca portal.utoronto.ca
App1!
App2!
App3!Web Login !(pubookie)!
!bbconfig.auth.type=ldap!!
!bbconfig.auth.type=toronto!!
F5
1 out of 4: Blackboard Mobile
Looking for service and performance improvements
Apache 2.2 • Feasible on SP5 yet hard to administer
Shibboleth • Possible with Apache 2.2 yet hard to administer
MinificaNon • Not worthwhile without compression
12
13
OCHO Looking forward to 9.1SP8
The Plan
14
Apache 2.2.x • SSL and Compression working together
Shibboleth • New Authentication Framework
Bb Mobile • Possible with Apache 2.2.x and Shibboleth?
Minification • Worthwhile without compression?
T4-4 • Consolidation and Cryptographic Acceleration
Target version: 2.2.2
15
• Modules
Apache2
16
Compilation
64bit Binary for SPARC
!
CC="cc -m64“ !
CXX="CC -m64“ !
CFLAGS="-m64 -xO2 -DSSL_ENGINE“ !
CXXFLAGS="-m64 -xO2“ !
LDFLAGS="-L/usr/sfw/lib/sparcv9 !
! ! -R/usr/sfw/lib/sparcv9“ !
CCFLAGS="-m64“ !
Shared Modules (DSO)
!
--enable-mem-cache=shared!
--enable-file-cache=shared!
--enable-headers=shared!
--enable-usertrack=shared!
--enable-expires=shared!
Read More
Apache2
Performance.conf
<IfModule mpm_worker_module>!
ServerLimit 1024!
StartServers 341!
MinSpareThreads 64!
MaxSpareThreads 128!
ThreadLimit 128!
MaxClients 1280!
ThreadsPerChild 128!
MaxRequestsPerChild 0!
</IfModule>!
!
Proxy_ajp.conf
<IfModule proxy_module>!
ProxyRequests Off!
ProxyTimeout 3600!
# Shibboleth !
ProxyPassMatch ^(/shib.*)$ !!
ProxyPass /Shibboleth.sso !!
ProxyPass /shibboleth-sp !!
ProxyPass /Shibboleth.sso/Status !!
</IfModule>!
!
!
17
Configuration
# Blackboard secure area !# This will ensure that mod_shib ignore all!# requests except those sent to !# .../execute/shibbolethLogin.!<Location /webapps/bb-auth-provider-shibboleth-bb_bb60/execute/shibbolethLogin>! AuthType shibboleth! Require shibboleth! ShibRequestSetting requireSession 1! Require affiliation ~ ^member@.+$! Require user ~ ^.+$! Require affiliation isstaff! Require affiliation isstudent!</Location>!
Apache2
18
Shibboleth Configuration
# Blackboard Mobile Learn B2 Configuration!# In older installations BBLEARN should be !# changed by bb_bb60!<Location /webapps/Bb-mobile-bb_bb60>! AuthType shibboleth! ShibRequestSetting requireSession 0! Require shibboleth! Require user ~ ^.+$! Require affiliation ~ ^member@.+$! Require affiliation isstaff!</Location>!!
Apache2
Compressed Weight
19
Total Weight
1036.9K 265.7K
Web Compression + SSL
74.3%
VS
vS
20
force to native
© Blackboard Mobile: h`p://help.blackboardmobile.com
web
Blackboard Mobile Learn Authentication Type
Shibboleth and LDAP
21
Implementing a New Authentication Framework
LDAP Server!
App4!
mobile.lms.utoronto.ca portal.utoronto.ca
App1!
App2!
App3!Shibboleth !2.4.3!
F5
Hostname RestricNon Provided by the New AuthenNcaNon Framework
Provider: Toronto Shibb Auth Provider: Toronto LDAP Auth
Shibboleth and LDAP
Shibboleth LDAP
22
Implementing a New Authentication Framework
mobile.lms.utoronto.ca portal.utoronto.ca
Minification
23
Real path MinificaNon MinificaNon CR MinificaNon t $BBHOME/docs 835,860.00 50.93% 3.1886 $BBHOME/webapps/blackboard 183,477.00 58.46% 0.6999 $BBHOME/webapps/assessment 51,225.00 58.01% 0.1954 $BBHOME/webapps/discussionboard 30,919.00 35.52% 0.1179 $BBHOME/webapps/gradebook 277,527.00 54.81% 1.0587 $BBHOME/webapps/caliper 119,764.00 48.56% 0.4569 $BBHOME/webapps/portal 27,595.00 54.81% 1.0600 $BBHOME/webapps/cms+xy 49,532.00 52.59% 0.1889 $BBHOME/webapps/wysiwyg 99,681.00 52.43% 0.3803 $BBHOME/webapps/webeq-‐plugin 15,354.00 52.43% 0.3800 $BBHOME/webapps/taglibs 44,054.00 52.43% 0.3800 $BBHOME/webapps/* 4,936.00 52.43% 0.3800 Grand Total 1734988.00 8.11
1694.32K MR ~ 52.0%
Savings on payload
Minification
• Prematurely released on SP5
• Released on SP8 as certified.
• Implementing YUI Compressor Library
• Grouping and minifying on-the-fly (inside JVM)
• Enabled by default on SP8
24
Blackboard JS Grouping Tool
## Whether related JavaScript files should be grouped together ## ## for be`er HTTP performance ## bbconfig.javascript.group.files=true
Read More.
<script type="text/javascript" src="/branding/__js__/C131DA0400D29916A81632A83B91BAD2.js?v=9.1.50119.0"></script>
25
Minification Blackboard Grouping Tool
Firebug console output sample
Read More.
26
Solaris Cryptographic Framework (SCF)
27
© Sun Microsystems: Using The Cryptographic Accelerators in the ULTRASPARC T1 and T2 Processors.
28
conf/pkcs11.conf
SSLCryptoDevice pkcs11!
64bit Binary for SPARC
!
CC="cc -m64“ !
CXX="CC -m64“ !
CFLAGS="-m64 -xO2 -DSSL_ENGINE“ !
CXXFLAGS="-m64 -xO2“ !
LDFLAGS="-L/usr/sfw/lib/sparcv9 !
! ! -R/usr/sfw/lib/sparcv9“ !
CCFLAGS="-m64“ !
Read more.
Linking Apache2 binary
Solaris Cryptographic Framework (SCF)
29
OCHO Current environment 9.1SP8
Infrastructure: Hardware
30
App1!
App2!
App3!
Hitachi 9985!2 RAID 6 Arrays!
2 TB x 7200 RPM SATA!Carved into 192 GB Ldev!
!
Collab1!
Data1!
SPARC T4-4!4 CPU @ 3.0 GHz!
8 cores/CPU!256 threads!512GB of RAM!
4 x LDOMs!24 vCPUs!
64G of RAM!
1 x LDOM!56 vCPUs!
120G of RAM!
Bb 9.1SP8
App4! 2 x LDOMs!24 vCPUs!
32G of RAM!App5!
31
Infrastructure: Hardware SPARC T4-4 and Oracle VM Server for SPARC v2.2
64 64 64 32 32 120 64 8
Collab1 App1 Ap2 App3 App4 App5 Data1 IO/Controller Free
Memory
32
Infrastructure: Hardware
32 32 32 16 16 56 32 24
Collab1 App1 Ap2 App3 App4 App5 Data1 IO/Controller Free
SPARC T4-4 and Oracle VM Server for SPARC v2.2
vCPUs
33
Infrastructure: Hardware Live Migration
Read More.
Source
Target
34
Infrastructure: Software Bb 9.1SP8
Blackboard Learn 9.1 SP8
Apache 2.2.2 64 bit SPARC
Shibboleth (DSO)
SSL
Apache 2.2.2 • Compression with SSL • SSL Offloading – PKCS11
Blackboard Mobile • NaNve AuthenNcaNon
AuthenNcaNon Providers:
LDAP + Shibbholeth
Shibboleth • LDAP
Performance
35
Benchmark
requests were sent sequentially with different concurrency levels 50k
Proxy SSL
AJP <
https://server/webapps/portal/healthCheck
deflate Concurrency Apache 2.2.2 Apache 2.2.2
SSL-‐H, AJP SSL-‐H, AJP, COM 10 1230.59 1143.12 100 1962.52 1704.3 200 1699.73 1625.22 500 1870.60 1075.2 1000 1214.95 1173.457 2000 1129.87 1234.44
@ 1k request/sec > process ~1.2K req
Performance
36
24%
56%
17%
2% 1% 0% 0% 0%
0 -‐ 1
1 -‐ 3
3 -‐ 7
7 -‐ 13
13 -‐ 21
21 -‐ 35
35 -‐ 60
60+
Load Times
Avg. Page Load Time:
2 .44 SEC
