Upload
others
View
14
Download
0
Embed Size (px)
Citation preview
Cisco SE TrainingJanuary 27, 2012
Laurent Gharda, [email protected] [email protected]
Cisco Server Provisioner 6.4
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 2
Background
Audience:Those familiar with TSP 6.3
Presentation Contents:What’s new in CSP 6.4Followed by prior TSP 6.3 training slides
This presentation deck can be given to people new to CSPStart with TSP 6.3, then move to CSP 6.4
Effective with this release:Old name: Cisco Tidal Server Provisioner (TSP)New name: Cisco Server Provisioner (CSP)
GUI and User’s Guide have been re-brandedPackage name still uses “tsp”, will until December 2012 release
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 3
CSP 6.4 TAC Training Agenda:Changes from TSP 6.3
Installation and UpgradeInstallation: new Network Preparation scriptUpgrade from TSP 6.3
New Features in CSP 6.4 (customer-visible)Provisioning RHEL/CentOS 6.xProvisioning and Imaging of multi-NIC clientsPost-provisioning booting (IP address for given VLAN)Listing of installed packages/scripts on Help/About
New Functionality (behind the scenes)Unknown client PXE-booting management
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 4
CSP 6.4 Resources
CSP Software: http://www.cisco.com/cisco/software/type.html?mdfid=284002585&catid=268439477
Release Notes (and much more) at http://www.linmin.com/site/tsp_release_notes.html
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 5
Installation
New Network Preparation scriptStart with properly configured networkingRun this script as the first part of the installation process
Then run Server Preparation Script
Avoids issues found with non-Linux, non-Network folksDifficulty using Linux editorsTransposing numbers, punctuation, values, etc.
Detects current network settings, prompts user for confirmation, automatically configures files that previously needed to be configured manually:
Ifcfg-eth0/eth1, network, resolv.conf, hosts
Reference: http://www.linmin.com/cisco_64/help/index.html?quick_start_guide.html“Prepare your System (including Examples of Network Configuration Files)”
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 6
Upgrade to CSP 6.4 from TSP 6.3
Place all files in /usr/local/linminCSP, WPM, EPM and patchsh cisco-tsp-6.4.0-6-p1.exp (CSP package is still labeled “tsp”)Existing installation of TSP 6.3 will be detected, you will be prompted to OK the upgradeFollow the instructionsReboot
Reference:http://www.linmin.com/cisco_64/help/index.html?quick_start_guide.html
“Installation and Upgrade”
Recommendation: that all TSP 6.3 sites be upgraded to CSP 6.4 as soon as possible
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 7
CSP 6.4 New Feature:Provisioning RHEL/CentOS 6.x
No change in process from provisioning RHEL/CentOS 5.xloadlinux.plCreate MAC-Specific Provisioning Role TemplateAssign Template to a MAC-Spec Provisioning Role
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 8
CSP 6.4 New Feature:Provisioning and Imaging Multi-NIC Clients
Provisioning:For ESX/ESXi, RHEL/CentOS, Windows Server 2008 & R2
Imaging: all OSsSupports up to 4 NICsFor non-UCS clients, discover MAC addresses by PXE-booting without network connectivity
Reference: http://www.linmin.com/cisco_64/help/index.html?client_systems_mac_address_multiple_nics.html
Enter MAC address in Prov. Role or Imaging ProfileNote for Windows Server 2008:
“Local Area Connection” is arbitrary (can be null, 2, 3 or 4)
IP can be assigned to any (Windows “feature”)
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 9
CSP 6.4 New Feature:IP assignments are based on VLAN
Implemented such that DHCP is not required on non-PXE (production) VLANs (major issue with many data centers)Behavior:
RHEL/CentOS and Window Server 2008 & R2:Upon reboot on PXE VLAN, Client uses the dynamic DHCP address during provisioning as long as it’s on PXE VLANUpon reboot on VLAN other than the PXE VLAN, Client switches to static IP address assigned in GUI or by TEO via API
ESX/ESXiUpon reboot on any VLAN (including PXE VLAN), IP will be set to the Production VLAN static IP set in GUI or by TEO via APIClient is unreachable by IP while on PXE VLAN and until Client is on correct production VLAN (Houston requested this implementation since TEO doesn’t access Clients on PXE VLAN)
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 10
CSP 6.4 New Feature:Help/About lists installed/used components
Easy way to ensure that customer has installed CSP correctly with all componentsInformation is captured in .help file
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 11
CSP 6.4 New FunctionalityUnknown MAC Address Management (1 of 3)
Background:With TSP 6.3, when the MAC-Indy feature is disabled and clients with unknown MAC addresses make PXE boot requests, a DHCP flooding "Denial of Service" can occur
TSP DHCP worked overtime, MAC addresses cluttered logs and were captured by the licensing mechanismThis led to unpredictable behaviors when switching from MAC-Indy to MAC-Spec and re-using the same Client
See the “Notes” page for this slide for more details
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 12
CSP 6.4 New FunctionalityUnknown MAC Address Management (2 of 3)
With CSP 6.4:When MAC-Indy is disabled, unknown MACs get logged and CSP presents a message to the client preventing the "Denial of Service“Client-side message is as follows:
...............................................................................###################################################################Welcome to the Bare Metal Provisioning and Imaging Service
No MAC-Specific Provisioning Role or Imaging Profilehas been activated for this NIC's MAC address (displayed above),or MAC-Independent Provisioning has not been enabledby your BMP System Administrator.
Within a few seconds, there will be an attempt to bootto the next designated boot device or PXE boot to thenext NIC of this system..................................................................................
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 13
CSP 6.4 New FunctionalityUnknown MAC Address Management (3 of 3)
If MAC-Indy is used first, then MAC-Spec is used with the same Clients (a non-production scenario, but is typical with POCs with SEs “getting their feet wet” with CSP), unexpected behaviors can still occur
See the “Notes” page for this slide for more detailsBest practices:
Take a VM snapshot before “playing” with MAC-IndyPlay with MAC-Indy
Revert to pre-MAC-Indy snapshot when ready to use MAC-Spec with GUI or TEO for true POC or production implementationDo not use the same MAC addresses with MAC-Indy and then with MAC-Spec
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 14
CSP 6.4 Updated Documentation
UCS Manager Blade Client Configurationhttp://www.linmin.com/cisco_64/help/index.html?cisco_ucs_clients.html
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 15
CSP 6.4: What’s coming next…
Replacement patchAbility to provision ESXi 5.0 (planned)New scripts for enhancing diagnostics process
Did PXE-boot reach the CSP Server or is there a network configuration issue?Database and log files search by MAC, by IP or for all valuesEtc. (more details forthcoming)
Support-help script to automatically execute diagnostic script(s) to include results in .help file
Will be made available from and supported directly by LinMin for select POCs and TAC until Houston QA vets and releases the patch
2012-1-12 © Copyright 2012 LinMin Corporation - www.linmin.com 16
CSP 6.4 TAC Training Wrap-up
Q&A
2011-7-14 © Copyright 2011 LinMin Corporation - www.linmin.com 18
Session 1 – October 5, 1:00 p.m. – 2:00 p.m.
Concepts of operationDemo of product functionalityDemo of product installation
Session 2: Self-Paced (Oct. 5, 2011)
Each Systems Engineer to:Install TSP 6.3 on a VM (on ESXi, not ESX)Provision at least 1 VM with:
CentOS 5.6 x86_64 or Windows Server 2008 R2If available, provision 1 blade with ESXiImage backup of the RHEL or R2 VM, then restore itGenerate a .help file
Estimated time:Installation: 1 hourProvisioning: 1.5 hours (load ISO, create Template+Role, provision)Imaging: ½ hour
2011-7-14 © Copyright 2011 LinMin Corporation - www.linmin.com 19
Session 3: Oct 5, 2011
Attendees must have performed the steps in Session 2TSP 6.3 architecture & file structureOverview of TSP 6.3 Troubleshooting
Installation issuesOperational issuesUnderstanding how to use .help files
Review of common support issues
2011-7-14 © Copyright 2011 LinMin Corporation - www.linmin.com 20
Cisco SE Boot Camp- Session 12011-10-5
Cisco Tidal Server ProvisionerConcepts of Operation
Why Automate Infrastructure Deployment?
Increase Business Agility & Reduce Elapsed Time, e.g.:Quickly get 5 to 500 new servers online for a customerLet customers re-provision systems 24x7 with no assistanceRepurpose systems from a prior customer for a new one
Predictability, Quality, RepeatabilityEach system will be exactly like the others (when desired)
Labor costs: no more time wasted building systems by handSys Admins should do skilled, not repetitive work
Scalability: grow your infrastructure w/o headcount growthSys Admins can be located anywhere & service many sites
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
22
TSP CapabilitiesIT Software that Controls the Pre-Operating System Domain
The Time Between Power-On and System Boot
Provisions (remotely installs) the Operating SystemWindows & Linux (and applications, scripts, files) & ESX/ESXiMAC-Specific Provisioning (for Production) and Mac-Independent (for Sandbox or PC deployments)Deployment of Rescue Systems into RAM for troubleshooting, recovery
Images (disk capture and restore of) Windows, Linux, ESXDeploys Rescue OSs
For Physical Systems and Virtual Machines
Managed via GUI (browser) or API (HTTP) (e.g., TEO)
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
23
System Provisioning at a Glance
• OS Media (ISO files)
• OS Role Templates
• IP & MAC Addresses
• Answer Files
• Scripts
• Applications
• Drivers
• Backup Disk Images
• Rescue Systems
TSP Server:
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
24
TSP Controls the Pre-OS Domain:The Time Between Power-On and System Boot
TSP Server
Data Center Servers
IT Applications• TEO
• Control Panels
• IT Automation
Browser
PXE, TFTP, HTTP, NFS
HTTP via API
TSP Business Rules upon Client power-on:
• Take no action (steady state): boot to disk
• Never Provision (fail safe)
• Provision just once, then switch to "Never"
• Perform full system (disk image) backup
• Perform full system restore ("roll back")
• Deploy a Rescue OS
PCs
Optional
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
25
Architecture
Runs on Linux: Red Hat Enterprise Linux or CentOS 5.xDatabase (PostgreSQL: systems, file locations, etc.)File System (for ISOs, images, control files, etc.)Web server (for GUI access)Low level services (TFTP, PXE, DHCP, HTTP, NFS)API (used for TEO integration)Languages:
GUI: PHPEngine: JavaScripts: PHP, perl, bash
2010-09-22
© Copyright 2010 LinMin Corp. www.linmin.com - Confidential 26
TSP relies on PXE:Pre-Boot eXecution Environment
Time
BIOSInitialization
BIOSInitialization
NICBoot ROM
NICBoot ROM
DHCPRequestDHCP
Request
Bootloader(from TFTP
server)
Bootloader(from TFTP
server)
BootMenuBootMenu
KernelAnd FSKernelAnd FS
PXE = DHCP + TFTP
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
27
Unattended OS Installation
Sample PXE Boot and Provisioning SequenceTSP Built-in Components: DHCP, TFTP, NFS, HTTP
22
3344
55
66
77
DHCP Server
TFTP Server
File Server (NFS, HTTP)
88
11
99TSP
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
28
Bare Metal Provisioning in 3 Steps
Repeatability, Accuracy, Continuity in Bare Metal System Deployment
OS:Red Hat®Windows®ESX®/ESXi®Ubuntu®CentOSDebianNovell® SLESFedoraRescue System CD
+PackagesFilesApplicationsConfigurationsScriptsImages
Upload into TSP: Deploy to Physical and Virtual Machines
Create “Templates” & "Roles" for "Push" &
"Pull"Bare Metal Provisioning
Steps: 1 2 3
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
29
TSP is a Sophisticated Solution:7 Unique Paths (for Provisioning alone)
TSP understands 7 fundamental OS/Hypervisor architecture “families”, each with unique installers, control files, methods, paths, etc.
Red Hat-based: RHEL, CentOS, Fedora“Modern Windows”: Windows Server 2008/R2, Win 7“Legacy Windows”: Windows Server 2003, Win XP“Modern hypervisor”: ESXi“Legacy hypervisor”: ESXDebian-based: Debian, UbuntuSUSE-based: Novell SLES, OpenSUSE
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
30
TSP Provisioning Method 1:MAC-Independent Provisioning (“MAC-Indy”)
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
31
MAC-Indy Provisioning: 2 Clicks to Create a Windows Server 2008 R2 Provisioning Role
Step 1:Select
Step 2:Done
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
32
TSP Provisioning Method 2:MAC-Specific Provisioning (“MAC-Spec”)
Used by TEO
Gives IT Precise Control Over Each SystemOS, apps, host name, password, IP, time zone, scripts, files
Lets IT Pre-Stage Systems for RepurposingSwitch systems to a dedicated provisioning VLAN then back to the Live/Customer VLAN
Lets IT Decide What Actions a System Takes:TSP to ignore the system, tell it to boot from local disk (Safe Mode)TSP to provision each time a system boots to the networkTSP to provision once, then go to "Safe Mode" (boot from HD)TSP to deploy Rescue OS to RAM: e.g., Live Ubuntu
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
33
MAC-Spec ProvisioningStep 1: Create Provisioning Role Templates
Step 1:Select
Step 2:Done!
The form is auto-filledwith path
to kernel & RAM disk,configuration file and
optional kernel parameters
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
34
MAC-Specific Provisioning:View All Available Provisioning Templates
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
35
MAC-Specific ProvisioningStep 2a: Assign Template to a System
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
36
MAC-Specific ProvisioningStep 2b: Assign Unique Settings to System
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
37
• Each system will be provisioned with a pre-determined profile (OS/hypervisor, web/app/DB server, workstation)
• Full hardware discovery is performed such that the client systems can have different CPU types, NICs, RAM, discs, etc.
MAC-Specific ProvisioningTSP Dashboard: All Systems at a Glance
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
38
2011-7-14 © Copyright 2011 LinMin Corporation - www.linmin.com 39
Deploying Rescue System OSs
For remote repair of systems: deploy to RAM, not to diskLive Ubuntu = Ubuntu + Clonezilla Live (disk imaging)Red Hat/CentOS Live
2011-7-14 © Copyright 2011 LinMin Corporation - www.linmin.com 40
API: Sample Code Generator Makes it EasyUsed by Tidal Engineering for TEO Adapter and Workflows
"Teaching Mode" generates working API code for easy implementation
2011-7-14 © Copyright 2011 LinMin Corporation - www.linmin.com 41
Imaging: Backup, Restore, CloneSystems at a Glance
2011-7-14 © Copyright 2011 LinMin Corporation - www.linmin.com 42
Imaging: a System Profile
• IT Staff Pre-configures snapshot/rollback profiles through the Browser-based GUI
• Each system will have all its partitions backed up and stored on the network
• Systems can be restored to a fully "known-good" point in time upon IT request
• Since the OS is not running during snapshots, full file/service/database integrity is ensured
• System-Specific (tied to MAC address) "Hardware Specific Backup/Restore" = Snapshot/Rollback
• Can be used to clone systems with identical hardware
File LocationsTSP housekeeping (installable packages, license key, support scripts, services, most logs):
/usr/local/linmin/Media and user management scripts:
/home/tftpboot/bin/ISO media:
/home/tftpboot/ISOs/ (then linux/, esx/, etc.)Installed media, customization files:
/home/tftpboot/pub/os_name/ or */install_instance_name/Reference:
Operational files: http://www.linmin.com/cisco_63/help/index.html?advanced_topics_file_locations.html
Log/Audit files: http://www.linmin.com/cisco_63/help/index.html?managing_your_provisining_server_audit_logs.html
2010-09-22
© Copyright 2010 LinMin Corp. www.linmin.com - Confidential 43
LicensingLicense key is tied to MAC address of the TSP Linux systemInstall key with 1 command: “sh filename.exp” http://www.linmin.com/cisco_63/help/index.html?managing_your_provisioning_server_update_the_license_key_file.htmll
License keys control: Expiration of keyMaximum number of provisioned/imaged clientsUse of optional modules
ESX Provisioning ModuleWindows (Server 2008/Win7) Provisioning Module
Reference: http://www.linmin.com/cisco_63/help/index.html?advanced_topics_license_keys_and_licensing.html
Cisco engineers specify what they need, e.g.:For internal use: long duration, lots of clients, all bellsFor trials: 30/45/90/? days, 10/50/? ClientsTo request a temporary license key (with a Cisco email address), complete the short form at http://www.linmin.com/site/contact_cisco_tsp.html
2010-09-22
© Copyright 2010 LinMin Corp. www.linmin.com - Confidential 44
TSP Concepts of Operation
Next:
Live TSP Demo &TSP Installation
Importance of the TSP User’s Guide:The User’s Guide (“UG”) is the combination of all TSP documentation and the
TSP Knowledge Base. Always refer to (or search) the User’s Guide when learning/using or supporting/troubleshooting TSP.
http://www.linmin.com/cisco_63/help/ then when 6.3 is released, also available athttp://www.linmin.com/cisco/help/
Before contacting TAC: http://www.linmin.com/cisco_63/help/index.html?troubleshooting_technical_support.html
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 10, 2011
45
Session 2: Self-Paced
Each Customer Support Engineer to:Install TSP on a VM (on ESXi, not ESX)Provision at least 1 VM with:
CentOS 5.6 x86_64 or Windows Server 2008 R2If available, provision 1 blade with ESXiImage backup of the RHEL or R2 VM, then restore itGenerate a .help file
Estimated time:Installation: 1 hourProvisioning: 1.5 hours (load ISO, create Template+Role, provision)Imaging: ½ hour
Resources:User’s [email protected], emergency mobile # and WebEx
2011-7-14 © Copyright 2011 LinMin Corporation - www.linmin.com 46
Cisco SE Boot Camp- Session 32011-10-5
Cisco Tidal Server Provisioner 6.3Deeper Dive and Troubleshooting
Sessions 3
Overview of TSP TroubleshootingInstallation issuesOperational issuesUnderstanding how to use .help files
Review of common support issues
2011-7-14 © Copyright 2011 LinMin Corporation - www.linmin.com 48
TSP 6.3 Training Session 3: Agenda
Review of Customer materials needed for effective supportTSP Configurations (2 VLANs vs. 1 VLAN)Server Preparation and Installation IssuesOverview of TSP file layout on live TSP server
TSP ServicesReview of file locations, useful scripts and understanding how to use a .help file (with openHelpPackage.sh script)
Common Operational Issues and how to troubleshootLicense key issuesClient not finding the TSP serverPXE IP and State Management (& “Force Free IP”)
Q&A, Wrap up and Self-Paced Next Steps© Copyright 2011 LinMin Corporation - www.linmin.com
August 24, 201149
Customer materials needed for effective support
The reference: http://www.linmin.com/cisco_63/help/index.html?troubleshooting_technical_support.html
High-level summary (for details, review the UG link above):Descriptions:
TSP EnvironmentFunction being performed (and method: GUI vs. API)ClientOverview of the issue, symptomsSteps to enable Support to reproduceReferences to User’s Guide used
Attachments:Screen shots.help file
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 24, 2011
50
TSP Configurations (2 VLANs vs. 1 VLAN)Reference: http://www.linmin.com/cisco_63/help/index.html?quick_start_guide.html
“Configuration Overview” section
2 VLANs:Recommended for TEO implementations1 NIC on 1 VLAN for GUI/API and Internet/repository access (1 IP needed)1 NIC on a dedicated PXE VLAN for Provisioning/Imaging
1 VLAN (“Range” model)1 VLAN is shared: 1) GUI/API access, 2) Provisioning/Imaging Clients and 3) unknown “guests” (PCs, other systems, etc.)Viable if the VLAN doesn’t have other unknown systems or DHCP serversRequires configuring non-TSP DHCP servers on VLAN
Non-TSP DHCP must forward PXE boot requests to TSP IP address and must exclude PXE IP range used by TSP.http://www.linmin.com/cisco_63/help/index.html?managing_your_provisioning_server_provisioning_server_configuration_dhcp_lan_vlan_considerations.html
Can be complex to troubleshoot network configuration issues/conflicts© Copyright 2011 LinMin Corporation - www.linmin.com
August 24, 201151
Server Preparation (and Installation) Issues
Reference: http://www.linmin.com/cisco_63/help/index.html?quick_start_guide.html
“Prepare your System” and “Installation” sections
Several potential pitfalls if Quick Start Guide (QSG) not followedAll pitfalls addressed by documentation in QSGPitfalls will be addressed in a replacement Server Prep script (delivery to Tidal in a few weeks) to detect/prevent:
Running on unsupported RHEL/CentOS 6.x and incorrect root accessPrevents execution of support-help script. Escalate to LinMin.
Duplicate entries in ifcfg-ethx filesSystem time not set (installing packages from the future), not overwrite RHN Satellite yum URL override, prevent running on 32-bit
Server prep provides clear diagnostics: customer to repeat running it until all conditions have been met.
If still unable to complete, customer to run script with option to create a zipped log file and email to Support
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 24, 2011
52
Training on a live TSP 6.3 Server:
Overview of TSP file layout on live TSP 6.3 serverReview of file locations, useful scripts and understanding how to use a .help file (with openHelpPackage.sh script)
TSP Serviceshttp://www.linmin.com/cisco_63/help/index.html?managing_your_provisioning_server_startstoprestart_the_provisioning_server.html
Common Operational Issues and how to troubleshootLicense key issuesClient not finding the TSP server
http://www.linmin.com/cisco_63/help/index.html?troubleshooting_clients.html
PXE IP and PXE Event State Management (& “Force Free IP”)http://www.linmin.com/cisco_63/help/index.html?advanced_topics_pxe_ip_address_pre_assignment.htmlhttp://www.linmin.com/cisco_63/help/index.html?advanced_topics_pxe_event_states.html
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 24, 2011
53
SE Boot Camp TSP 6.3 October 5, 2011
Q&AWrap upNext Steps
Self-paced exercises for SEs:Provision the OS/Hypervisors (RHEL/R2/ESXi) not done in Session 2Review UG top down (start with top, use “Next” arrow until done)Exercise TSP as driven by TEOReview Session 3 WebEx recording for the “Live TSP System” training, become comfortable with file locations, file interactions, scripts, etc.Install TSP in a single VLAN environmentProvision secondary Operating Systems (W2K3, Ubuntu, SLES)
WebEx Recordings of TAC sessions can be found at:http://wikicentral.cisco.com/display/TEOCOMM/TEO+2.2.0+Release+Training
© Copyright 2011 LinMin Corporation - www.linmin.comAugust 24, 2011
54