Upload
aestas-it
View
447
Download
4
Embed Size (px)
Citation preview
01
About me02
Andrey AdamovichBio: Developer, coach, speaker, author
Company: Aestas/IT (http://aestasit.com)
Email: [email protected]
Linkedin: http://www.linkedin.com/in/andreyadamovich
Lanyrd: http://lanyrd.com/profile/andreyadamovich
GitHub: https://github.com/aadamovich
SO: http://stackoverflow.com/users/162792/andreyadamovich
Twitter: @codingandrey, @aestasit
••••••••
03
What's this presentation about?DevOps
Intrastructure Provisioning
Continuous Integration
Continuous Delivery
••••
04
TechnologiesGroovy http://groovy.codehaus.org
Gradle http://gradle.org
Jenkins http://jenkinsci.org
Puppet http://puppetlabs.com
AWS http://aws.amazon.com
•••••
05
Developers +Operations =
?06
First Blood07
Ant + Gradleant.taskdef(
name: 'scp',
classname: 'o.a.t.a.t.o.ssh.Scp',
classpath: configurations.secureShell.asPath)
ant.taskdef(
name: 'sshexec',
classname: 'o.a.t.a.t.o.ssh.SSHExec',
classpath: configurations.secureShell.asPath)
01.
02.
03.
04.
05.06.
07.
08.
09.
08
Simple callant.sshexec(
host: host,
username: user,
password: password,
command: command,
trust: 'true',
failonerror: failOnError)
01.
02.
03.
04.
05.
06.
07.
09
Next step: wrapper functiondef ssh(String command,
Properties props,
boolean failOnError = false,
String suCommandQuoteChar = "'",
String outputProperty = null)
...
01.
02.
03.
04.
05.
06.
07.
10
Next step: wrapper functiondef scp(String file,
String remoteDir,
Properties props)
...
01.
02.
03.
04.
05.
11
Task example Itask installFonts <<
forAllServers props ‐>
ssh('yes | yum install *font*', props)
01.
02.
03.
04.
05.
12
Task example IItask uninstallNginx <<
forAllServers props ‐>
ssh('/etc/init.d/nginx stop', props)
ssh('yes | yum remove nginx', props, true)
ssh('rm ‐rf /etc/yum.repos.d/nginx.repo', props)
ssh('rm ‐rf /var/log/nginx', props)
ssh('rm ‐rf /etc/nginx /var/nginx', props)
01.
02.
03.
04.
05.
06.
07.
08.
09. 13
DrawbacksNew connection each time
Excplicit repeating parameters
Complex scripts are hard to maintain
Tasks are not idempotent
••••
14
Sshoogr
15
Sshoogr featuresGroovybased SSH DSL for:
Remote command execution
File uploading/downloading
Tunneling
•••
16
Sshoogr usage (import)@Grab(
group='com.aestasit.infrastructure.sshoogr',
module='sshoogr',
version='0.9.16')
import static com.aestasit.ssh.DefaultSsh.*
01.
02.
03.
04.
05.
17
Sshoogr usage (defaults)defaultUser = 'root'
defaultKeyFile = new File('secret.pem')
execOptions
verbose = true
showCommand = true
01.
02.
03.
04.
05.
06.
18
Sshoogr usage (connection)remoteSession
url = 'user2:654321@localhost:2222'
exec 'rm ‐rf /tmp/*'
exec 'touch /var/lock/my.pid'
remoteFile('/var/my.conf').text = "enabled=true"
01.
02.
03.
04.
05.
06.
19
Sshoogr usage (multiline content)remoteFile('/etc/yum.repos.d/puppet.repo').text = '''
[puppet]
name=Puppet Labs Packages
baseurl=http://yum.puppetlabs.com/el/
enabled=0
gpgcheck=0
'''
01.
02.
03.
04.
05.
06.
07.
20
Sshoogr usage (file copying)remoteSession
scp
from localDir "$buildDir/application"
into remoteDir '/var/bea/domain/application'
01.
02.
03.
04.
05.
06.
21
Sshoogr usage (command result)def result = exec(command: '/usr/bin/mycmd',
failOnError: false, showOutput: false)
if (result.exitStatus == 1)
result.output.eachLine line ‐>
if (line.contains('WARNING'))
throw new RuntimeException("Warning!!!")
01.
02.
03.
04.
05.
06.
07.
08.
09. 22
Sshoogr usage (shortcuts)if (ok('/usr/bin/mycmd'))
...
if (fail('/usr/bin/othercmd'))
...
01.
02.
03.
04.
05.
06.
23
Sshoogr usage (tunnels)tunnel('1.2.3.4', 8080) int localPort ‐>
def url = "http://localhost:$localPort/flushCache"
def result = new URL(url).text
if (result == 'OK')
println "Cache is flushed!"
else
throw new RuntimeException(result)
01.
02.
03.
04.
05.
06.
07.
08.
09. 24
Sshoogr usage (prefix/suffix)prefix('sudo ')
exec 'rm ‐rf /var/log/abc.log'
exec 'service abc restart'
suffix(' >> output.log')
exec 'yum ‐y install nginx'
exec 'yum ‐y install mc'
exec 'yum ‐y install links'
01.
02.
03.
04.
05.
06.
07.
08.
09. 25
Still problemsComplex scripts are still not easy to maintain
Scripts are usually not idempotent••
26
Puppet27
Why Puppet?More mature than competition
Large community
Readable DSL
Good acceptance from DEVs and OPs
No need to learn Ruby ;)
•••••
28
Puppet example
29
Puppet provisioning
30
Puppet provisioning
31
Puppet provisioning
32
Puppet provisioning
33
Puppet state management
34
Puppet state management
35
Puppet state management
36
Puppet modules
37
Puppet modules
38
Puppet modules
39
Deploymentvs.
Infrastructuremanagement40
Push vs. Pull41
Pull
42
Push
43
Sshoogr +Gradle +Puppet44
Upload modulestask uploadModules <<
remoteSession
exec 'rm ‐rf /tmp/repo.zip'
scp
from localFile "$buildDir/repo.zip"
into remoteDir "/root"
...
01.
02.
03.
04.
05.
06.
07.
08.
45
Upload modules ...
exec 'rm ‐rf /etc/puppet/modules'
exec 'unzip /tmp/repo.zip ‐d /etc/puppet/modules'
01.
02.
03.
04.
05.
46
Apply manifeststask puppetApply(dependsOn: uploadModules) <<
remoteSession
scp
from localFile "$buildDir/setup.pp"
into remoteDir "/tmp"
exec 'puppet apply /tmp/setup.pp'
01.
02.
03.
04.
05.
06.
07.
08.
09. 47
What we solved?Separated infrastructure state description and operations tasks
Scripts became more maintainable and idempotent••
48
In the meanwhile...We started developing complex/generic Puppet modules
Modules need proper testing
...on different platforms
•••
49
Do you test, right?How to test this stuff?
How to reuse a JUnit approach to testing?
We wanted things to be SIMPLE!
•••
50
PUnit
51
PUnitSimple testing tool for verifying remote server state
Uses Sshoogr and JUnit
Reuse reporting features of JUnit
As simple as ...
••••
52
PUnit example (derby)class DerbyInstallTest
extends BasePuppetIntegrationTest
@Before
void installDerby()
apply("include derby")
...
01.
02.
03.
04.
05.
06.
07.
08.
53
PUnit example (derby)@Test
void ensureDerbyRunning()
command('service derby status > derbystatus.log')
assertTrue fileText("/root/derbystatus.log")
.contains('Derby')
assertTrue fileText("/root/derbystatus.log")
.contains('is running.')
01.
02.
03.
04.
05.
06.
07.
08.
54
PUnit example (derby)@Test
void ensureCanConnect()
Thread.sleep(10000)
uploadScript()
command('/opt/derby/db‐derby‐10.9.1.0‐bin/bin/ij ' +
'testDataScript.sql > derbytest.log')
...
01.
02.
03.
04.
05.
06.
07.
55
Continuous integration
56
Jenkins build
57
Nextproblem?
58
ScalabilityHow do we test on different OS?
How do we run parallel tests on multiple architectures?
How do we avoid selling our houses?
•••
59
Amazon WebServices
60
Elastic Compute CloudMature
Great API
Virtual hardware variety
OS variety
••••
61
Gramazon
62
GramazonGroovybased API for interacting with EC2
Integration with Gradle••
63
Gramazon example Itask startInstance(type: StartInstance)
keyName 'cloud‐do'
securityGroup 'cloud‐do'
instanceName 'gramazon/cloud‐do'
stateFileName 'cloud‐do.json'
ami 'ami‐6f07e418'
instanceType 't1.micro'
waitForStart true
01.
02.
03.
04.
05.
06.
07.
08.
09. 64
Gramazon example IItask terminateInstance(type: TerminateInstance)
stateFileName 'cloud‐do.json'
01.
02.
03.
65
The flowStart instance(s)
Upload manifests
Run tests
Generate report
Terminate instance(s)
1.
2.
3.
4.
5.
66
Next issue?67
Imgr
68
ImgrA tool for building images
Inspired by Packer••
69
SupportsShell
Puppet••
70
Configuration example
71
Summary72
Images, manifests, tasks
73
The big picture
74
ConclusionsGroovy/Gradle is an ultimate automation glue!
Reuse your existing Java knowledge
...to build a bridge between DEVs and OPs
Reuse development best practices for OPs
Infrastructure configuration is outsourcable now!
Automate!
••••••
75
Source codeSshoogr: https://github.com/aestasit/sshoogr
Sshoogr Gradle: https://github.com/aestasit/sshoogrgradle
Groowin: https://github.com/aestasit/groowin
Groowin Gradle: https://github.com/aestasit/groowingradle
PUnit: https://github.com/aestasit/puppetunit
Gramazon: https://github.com/aestasit/gramazon
Imgr: https://github.com/aestasit/imgr
•••••••
76
Questions?77
Thank you!78