Upload
mmik-huang
View
245
Download
2
Embed Size (px)
Citation preview
2014 AWS RE:INVENT和我們的故事 Mmik Huang
MMIK HUANG
6 years in Trend, 2 years in IT and 4 years in QA 12 years IT industry 2014 project: SecureCloud disk encryption service @ AWS Current project: Deep Security @ Azure
PAIN POINT FOR RD/QA
在過去 4 年的測試經驗中,發現最花時間的幾件事情有以下幾點 In the previous 4 years experience, the pain point from RD/QA
New feature/OS support. Prepare New OS template
Keep system up to date Lab testing environment GA Deploy
Collect log
Cloud Service 可以幫助你解決的問題有哪些 ? What kind of cloud services you can use ?
新版本的作業系統 New version of OS
自動化服務 Automatically services
有哪些 Cloud 我們有使用過 ? What kind of cloud we have experience?
Aws (Xen) Azure (hyper-v) ETS (vCloud) Terremark (cloudstack) Hpcloud (openstack) Rackspace (openstack)
試試看 AWS 吧 !!! Try AWS !!!
極低的入門門檻 Easy to leverage
Free tier, easy to pay Free support SQL Services, Auto scale Services, monitoring Services……
CLOUD JOURNEY
如果已經有使用 Cloud 的經驗,想要改善現有的測試架構,要如何開始 ?
換個方式問 : 如果你想蓋個房子,有哪幾種選擇 ? What is the first thing you will do, if you want to build a house?
方案 1: 直接動作做,大不了拆了重蓋 ? 方案 2: 聽聽看別人怎麼做 ? Option1: just do it~ Option2: get advice from expert.
以宅宅工程師的個性,方案 1 !!! As a geek, option1 !!!
做中學,從錯誤中學到經驗 Learning from failure is a good thing.
不過,如果是蓋 300 億的巨蛋呢 ? But if you want to build a 30 billion Taipei Dome?
方案 2 !!! Option2 !!!
方案 1 跟方案 2 的差別在於哪裡 ? What is the different between option1 and option2?
架構規劃 Good planning before starting
如果你想參考別人在 AWS 的架構,要去哪裡獲得呢 ? If you want to get advice for AWS services, where you can go?
AWS Re:Invent !!!
Learning AWS 2014 Re:Invent on Youtubehttps://www.youtube.com/channel/UCd6MoB9NC6uYN2grvUNT-Zg
Slide sharehttp://www.slideshare.net/AmazonWebServices/presentations
AWS 2014 RE:INVENT
MLB 7TB of data per game, 2340 games per session = 17PB of raw
data How will statcast change baseball (5:21) https://www.youtube.com/watch?v=847HY-JATrs
SUCCESS STORIES
Yelp http://www.yelp.com/
http://www.businessinsider.com/pacific-trust-thinks-yelp-is-deteriorating-2015-4
Map of restaurants S3: 1.2TB log per day Hadoop
MapReduce job: 30TB / 250 jobs
SUCCESS STORIES
SUCCESS STORIES
AdRoll https://www.adroll.com/
Provide CRM Service S3: 1.5PB Physical Servers
SUCCESS STORIES
Kellogg http://
www.kelloggs.com/en_US/home.html Food provider SAP HANA on AWS S3: 16TB
SUCCESS STORIES
Novartis https://www.adroll.com/
EC2: 10,600 spot instance Datacenter cost down (Own -> Rent) 39 years data under 9 hours
另外,聽聽看我們使用 AWS 的故事 Here is our story
出發前,當時測試環境的狀態 嘗試使用 AWS 做測試環境, beta site , staging site 嘗試透過 AWS 作環境自動化 嘗試建立 log collect system
The status before AWS Re:invent Try to use AWS as test lab, beta site and staing server Try to use 3rd party tool to build environment automatically Survey and build log collect system.
出發前收到 AWS 2014 Re:invent 的重點項目 The hightlights of AWS 2014 Re:invent
Service App development & mgmt. Big data & DR Security
AWS development tool All kind of architecture Success stories in different view AWS infra enhance
了解自己的需求,跟看過重點項目後 記得出發前的第一件事情是 Remember, the first thing before going to AWS Re:invent is
針對自己需求選課 !!!! ,四天可以聽的東西實在多到爆炸… Pick sessions online, there are too many thing need to share in 4 days
很多想聽的會衝堂,要提早做決定 Sometimes, the sessions you want to go will start in the same time, you need
to pick one
最好提早一天到飯店,到的時候放完行李,就去會場逛逛,熟悉一下 Go to seminar before AWS Re:Invent, check the map and the room location
before you go
早起的鳥兒 AWS 有得吃,準時開動準時開課,記得帶個水瓶去,沙漠水很貴 Get up early, you can get breakfast from AWS ($0, $28.99) Water is free in AWS, but not free outside
人實在太多,很多資源都是有限的,座位,食物,正妹解說員 : P Attend to any session on time, everything is limited, food , seats, even
representative : P
這次參加 AWS Re:Invent 最大的收穫,不是上課,是認識人 問講師問題,快速判斷哪些自動化服務可用,跟講師留聯絡資訊 吃飯時跟旁邊其他的公司員工打屁,順便 promote 一下自己的產品 去 Booth 看哪些廠商的產品可以試用看看
The best experience from AWS Re:Invent is not the sessions, is …. Asking questions to presenter, learning failure experience from them Talking to the people seat with you, lunch with you, get some
feedback related to your own product Going to booth and survey what you can use on the market
要認識人,記得要帶名片,還有準備一個 twitter account Bring your name cards there!!! As possible as you can, left email to 3rd party
venders and get evaluate license for service you are interesting. People use twitter there, not facebook, not line, not skype
四天吸收足夠的最新資訊後,就可以決定一下… .. After 4 days, you will learn a lot of new technical knowledge
要賭一把大的,還是回家繼續當宅宅工程師 You can decide to use it in casino, or go home
回去後,運用新知識,做出測試環境自動化 把小範圍測試,擴大測試範圍到 100 個 OS 把 python code 移植到 aws 利用 chef 做到 auto launch/purge vm 利用 Sumologic 蒐集 log
After back home, we use knowledge to enhance QA testing
Extend OS coverage to 100 Use the same python code to test AWS, ETS lab vm Use chef to build environment on AWS Use Sumologic as log collect system
改善環境自動化後,對我們的效益 Benefit for our product
過去,非常多的地方需要 QA 花時間,人力下去準備 Too many human resource (QA) manually trigger script, maintain environment
Phrase1 prepare environment [QA] Launch AWS instance [QA] install staff [QA] Setup related environment information (hosts, DNS, shell) [QA] Save as template [QA] Study AWS api, write code to handle instance (start,stop)
Phrase 2 trigger test [QA] Launch AWS instance [QA] execute python + staff
[staff] Deploy testing code [staff] Trigger testing and collect result/log
Phrase 3 get log and test result [QA] check instance one by one, cost high for investigate
經過環境自動化改善後,每個階段都大幅減少人力的時間 After leverage new knowledge, we can do things smart than before
Phrase1 prepare environment [AWS] leverage default aws template, no need to customize
Phrase 2 trigger test [QA] Launch AWS instance [Chef] execute python + staff
[staff] Deploy testing code [staff] Trigger testing and collect result/log
Phrase 3 get log and test result [Sumologic] collect log to Lab server [QA] check log, reproduce fail case
並且讓品質測試更穩定,涵蓋率更高,需要的人力更少 Product background
Support OS template on AWS: 58 (windows/linux) Support AWS 11 regions (different region with different AWS template) Aws will renew template every 6 months AWS templates/regions will increase day by day
Old day: Prepare 58 template every 6 months (launch instance, implement staff, save as template) Maximum coverage will be 638 templates (58*11) * ? hours Can not collect log back to lab
New way: No need to prepare template, use default AWS template Using chef to launch instance in 15 sec then deploy staf/python code to client Collect log back to lab for investigate.
新架構的好處 Benefit for new test framework
Benefit – easy scale out CSP (HP/Azure/AWS/vCloud…….) Zone (public/VPC/US/JP…..) OS (RHEL/Cent/uBuntu….)
Benefit – easy adopt current test script You can move any shell command into chef
Benefit – efficient testing for 1 instance from [1 day ] to [10-15 min]
Benefit – Cost No storage cost for keep customized templates
Benefit – easy investigate reduce cost for keep AWS vm running Centralized console for log search
讓品質更好,不一定要全部都自己來 在 AWS Re:invent 的 Booth 認識了 Chef 的 TAM ,他幫我們搭建了跟工程師的橋梁 淘汰了 Puppet 找到了三家 collect log 的廠商,在後來的 POC 決定用 sumologic
REFERENCE
如果聽完後,如果你對 chef 有興趣 If you are interesting about chef software
Create your first cookbook http://learn.getchef.com/legacy/tutorials/create-your-first-cookbook/
Learning Chef on Youtube http://www.youtube.com/channel/UCxEieNpB_tXiUBoF9zkPmAw http://nathenharvey.com/blog/2012/12/06/learning-chef-part-1/
Example http://gettingstartedwithchef.com/first-steps-with-chef.html
如果聽完後,如果你對 sumologic 有興趣 If you are interesting about sumologic software
Sumologic official site https://www.sumologic.com/
如果你正要去 Las Vegas If you are interesting about Las Vegas
Vegas bus
http://www.lasvegas-how-to.com/strip-downtown-express.php Show
https://www.caesars.com/ballys-las-vegas/shows/jubilee#.VE3YavmUd8G Buffet
http://www.totalrewards.com/microsites/las-vegas/buffet-of-buffets/
Q&A