Embed Size (px)
Citation preview
Solve the paradox
Less Downtime – More SecurityLinuxCon Berlin, Germany
October 4, 12:10 – 13:00
Hannes Kühnemund
SUSE Product Management
Downtime
Considerations for your digital architecture
Take a holistic approach …
- End-users (Business) are interested service availability
- Application, OS, Cluster, VM, Server, Network, Storage, People, Processes...
... because we understand that components will fail, ...
- Failure tolerant architecture, identify weak links
... acceptance of any downtime is decreasing and it is critical to ...
- Seek to reduce both planned and unplanned service downtime
... strike a balance.
- Cost of IT continuity vs. business impact
2
Downtime Quiz
Regular cadence
- monthly
- quarterly
- yearly
On the weekend
In alignment with all stakeholders
Combination of Taks
- software updates / configuration
- hardware exchange of defect parts
- datacenter maintenance / AC
Optimizable with
- SUSE Manager3
planned
Downtime Quiz
Regular cadence
- monthly
- quarterly
- yearly
On the weekend
In alignment with all stakeholders
Combination of Taks
- software updates / configuration
- hardware exchange of defect parts
- datacenter maintenance / AC
Optimizable with
- SUSE Manager4
planned unplanned
Downtime Quiz
Regular cadence
- monthly
- quarterly
- yearly
On the weekend
In alignment with all stakeholders
Combination of Taks
- software updates / configuration
- hardware exchange of defect parts
- datacenter maintenance / AC
Optimizable with
- SUSE Manager5
planned
No cadence
unplanned
Downtime Quiz
Regular cadence
- monthly
- quarterly
- yearly
On the weekend
In alignment with all stakeholders
Combination of Taks
- software updates / configuration
- hardware exchange of defect parts
- datacenter maintenance / AC
Optimizable with
- SUSE Manager6
planned
No cadence
Usually on Christmas Day
unplanned
Downtime Quiz
Regular cadence
- monthly
- quarterly
- yearly
On the weekend
In alignment with all stakeholders
Combination of Taks
- software updates / configuration
- hardware exchange of defect parts
- datacenter maintenance / AC
Optimizable with
- SUSE Manager7
planned
No cadence
Usually on Christmas Day
No alignment with stakeholders
unplanned
Downtime Quiz
Regular cadence
- monthly
- quarterly
- yearly
On the weekend
In alignment with all stakeholders
Combination of Taks
- software updates / configuration
- hardware exchange of defect parts
- datacenter maintenance / AC
Optimizable with
- SUSE Manager8
planned
No cadence
Usually on Christmas Day
No alignment with stakeholders
Only one particular problem fixed
unplanned
Downtime Quiz
Regular cadence
- monthly
- quarterly
- yearly
On the weekend
In alignment with all stakeholders
Combination of Taks
- software updates / configuration
- hardware exchange of defect parts
- datacenter maintenance / AC
Optimizable with
- SUSE Manager9
planned
No cadence
Usually on Christmas Day
No alignment with stakeholders
Only one particular problem fixed
Optimizable with
- Various technologies available
unplanned
Minimize Unplanned Downtime
10
Load Balancer
RAIDVirtualization
UPS
RASSystem
Rollback
High Availability
and GEO
Live Patching
Strike the balance?
11
Strike the balance?
12
No Downtime Security
13
But what about the non-disclosed
ones?
Since 2005, more than 75 data
breaches in which 1,000,000 or
more records were compromised
have been publicly disclosed.
Vulnerabilities
14
Year # vulnerabilities
2010 4258
2011 3532
2012 4347
2013 4794
2014 7038
2015 8822
2000
4000
6000
8000
10000
2010 2011 2012 2013 2014 2015
38%
16%18%
28%
Vulnerability type 2015
OperatingSystem
Browsers
Mobile Devices
Applications
Rank Operating System # vulnerabilities 2015
1 Apple OS X 384
2 Microsoft Windows Server 2012 155
3 Canonical Ubuntu Linux 152
4 Microsoft Windows 8.1 151
...
11 The Linux Kernel 77
Source: [http://www.cvedetails.com] & [https://nvd.nist.gov/] & [http://www.gfi.com/blog/2015s-mvps-the-most-vulnerable-players/]
15
In a data center, not so long ago …
In a data center, not so long ago …
16
Linux Kernel
Nov-11, 2015
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
17
Linux Kernel
Nov-11, 2015
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
18
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
Linux Kernel
Nov-11, 2015
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE: Common Vulnerabilities and Exposures
It is a standard naming scheme used by the NVD
NVD: National Vulnerability Database (https://nvd.nist.gov/)
In a data center, not so long ago …
19
Linux Kernel
Nov-11, 2015
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
20
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
Reboot
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
21
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE-2016-0728 CVE-2016-0728
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
22
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE-2016-0728 CVE-2016-0728
Reboot
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
23
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
CVE-2013-7446
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2016-0728
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2016-0728 CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
24
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
CVE-2013-7446
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2016-0728
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2016-0728 CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
Reboot
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
25
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
CVE-2013-7446
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0774
CVE-2016-2384
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0774
CVE-2016-2384
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-2384
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-2384
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
26
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
CVE-2013-7446
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0774
CVE-2016-2384
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0774
CVE-2016-2384
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-2384
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-2384
Reboot
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
27
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
CVE-2013-7446
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2016-1583
CVE-2016-3134
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
28
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
Linux Kernel
Jun-09, 2016
CVE-2013-7446
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2016-1583
CVE-2016-3134
Reboot
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
29
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
Linux Kernel
Jun-09, 2016
CVE-2013-7446
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2016-4997
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2016-4997
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2016-4997
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2016-4997
CVE-2016-1583
CVE-2016-3134
CVE-2016-4997
CVE-2016-4997
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
30
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
Linux Kernel
Jun-09, 2016
Linux Kernel
Aug-16, 2016
CVE-2013-7446
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2016-4997
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2016-4997
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2016-4997
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-1583
CVE-2016-2384
CVE-2016-3134
CVE-2016-4997
CVE-2016-1583
CVE-2016-3134
CVE-2016-4997
CVE-2016-4997
Reboot
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
31
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
Linux Kernel
Jun-09, 2016
Linux Kernel
Aug-16, 2016
CVE-2013-7446
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-0758
CVE-2016-1583
CVE-2016-2053
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-5829
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
32
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
Linux Kernel
Jun-09, 2016
Linux Kernel
Aug-16, 2016
Linux Kernel
Sep-12, 2016
CVE-2013-7446
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-0758
CVE-2016-1583
CVE-2016-2053
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-5829
Reboot
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a data center, not so long ago …
33
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
Linux Kernel
Jun-09, 2016
Linux Kernel
Aug-16, 2016
Linux Kernel
Sep-12, 2016
CVE-2013-7446
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-6480
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-6480
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-6480
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-6480
CVE-2016-0758
CVE-2016-1583
CVE-2016-2053
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-6480
CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-6480
CVE-2016-6480
CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-5829
CVE-2016-6480
Sample data taken
on Sept-15, 2016
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
That reminds me of ...
34
CVEs...? So what...?
• CVE-2016-0728
‒ gain privileges or cause a denial of service
• CVE-2015-8660
‒ local users can bypass intended access restrictions
• CVE-2015-8539
‒ gain privileges or cause a denial of service
• CVE-2015-7990
‒ allows local users to cause a denial of service
• CVE-2015-7872
‒ local users can cause a denial of service (OOPS)
• CVE-2015-6937
‒ local users can cause a denial of service (NULL pointer dereference and system crash)
• CVE-2013-7446
‒ local users to bypass intended AF_UNIX socket permissions or cause a denial of service (panic)
• ...
35
36
Can’t we patch
software while it runs?
Mankind already flew
to the moon …
Dynamic Software Updates
Trinity Test 1945 (Manhattan Project)
• IBM punch card automatic calculators
were used to crunch the numbers
• A month before the Trinity nuclear
device test, the question was: “What will
the yield be, how much energy will be
released?”
• The calculation would normally take
three months to complete –
recalculating any batches with errors
• Multiple colored punch cards introduced
to fix errors in calculations while the
calculator was running
37
kpatch
Modern history of kGraft and other DSU technologies
• DSU: Dynamic Software Updates
• the goal is to be able to fix bugs and add features either by
- changing some functions or
- replacing the whole program
• kGraft developed as Open Source project by SUSE Labs
• Upstream project „klp“
• Takes best of both kGraft (SUSE) and kpatch (Red Hat)
• Still in catch up w.r.t. to features required by enterprises
38
1990 201520001995 2005 2010
PoDUS Gupta Erlang Ginseng
UpStare
Ksplice Kitsune kGraftklp
ftrace: return address modification mechanism
39
Common Pitfalls
• Function Inlining
→ DWARF to the rescue
• Static Symbols
→ kernel keeps list: kallsyms
• IPA-SRA (optimization like -O2)
→ using gcc optimization log
• Multiple functions / dependencies
→ consistency model
• Eternal sleepers (getty console 10)
→ send fake signal SIGKGRAFT / ignore
• State transformation (req. for complex fixes)
→ not in kGraft right now
• 3rd party kernel modules
→ depends on what the module does ...
40
Consistency
Requirement: ensure system consistency when deploying live patches
41
Freezing the system (kpatch, ksplice) Lazy migration (kGraft)
Consistency
Requirement: ensure system consistency when deploying live patches
42
Freezing the system (kpatch, ksplice) Lazy migration (kGraft)
stop_kernel();
check all stacks, whether any thread is stopped within a patched function
If yes, resume kernel and try again later
If not, flip the switch on all functions and resume the kernel
Consistency
Requirement: ensure system consistency when deploying live patches
43
Freezing the system (kpatch, ksplice) Lazy migration (kGraft)
stop_kernel();
check all stacks, whether any thread is stopped within a patched function
If yes, resume kernel and try again later
If not, flip the switch on all functions and resume the kernel
For each thread separately:
Present the old version of functions to the thread until it leaves the kernel then give it the updated version
Wake sleeping threads up by a special signal Prevent the signal from reaching userspace
Once all threads have exited the kernel at least once we're DONE
Consistency
Requirement: ensure system consistency when deploying live patches
44
Freezing the system (kpatch, ksplice) Lazy migration (kGraft)
stop_kernel();
check all stacks, whether any thread is stopped within a patched function
If yes, resume kernel and try again later
If not, flip the switch on all functions and resume the kernel
For each thread separately:
Present the old version of functions to the thread until it leaves the kernel then give it the updated version
Wake sleeping threads up by a special signal. Prevent the signal from reaching userspace
Once all threads have exited the kernel at least once we're DONE
Do you have better ideas than those two? Join SUSE as Live Patching developer
https://jobs.suse.com/job/prague/live-patching-developer/3486/2529381
Consistency model for KLP?
The chosen model is a merge of kpatch and kGraft
• Combines stack checking and per-thread changes
• Non-intrusive, fast finishing
• Works well already but requires both:
45
Reliable stack unwinder (needed by kpatch)
• Worked on by Josh Poimboeuf @ Red Hat
• Currently needs FRAME POINTER
• up 10% slowdown of kernel execution
• Could use DWARF
• complex, being developed by SUSE
• speed is a concern
• initial implementation removed from
upstream
→ Takes time
Kernel thread model cleanup (needed by kGraft)
• Worked on by Petr Mladek @ SUSE
• Touches both kthreads and workqueues
• These parts are the critical core
• Needs a lot of good planning and review
→ Takes time
Live Patching on ppc64le?
46
[ http://mpe.github.io/posts/2016/05/23/kernel-live-patching-for-ppc64le/ ]
47
In a SUSE data center, today ;-)
In a SUSE data center, today ;-)
48
Linux Kernel
Nov-11, 2015
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
49
Linux Kernel
Nov-11, 2015
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
50
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
51
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
CVE-2016-0728
CVE-2016-0728
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
52
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
53
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660 CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660 CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
54
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
55
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-2384 CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-2384 CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-2384 CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0774
CVE-2016-2384
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
56
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
57
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
CVE-2016-1583
CVE-2016-3134
CVE-2016-1583
CVE-2016-3134
CVE-2016-1583
CVE-2016-3134
CVE-2016-1583
CVE-2016-3134
CVE-2016-1583
CVE-2016-3134
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
58
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
Linux Kernel
Jun-09, 2016
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
59
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
Linux Kernel
Jun-09, 2016
CVE-2016-4997
CVE-2016-4997
CVE-2016-4997
CVE-2016-4997
CVE-2016-4997
CVE-2016-4997
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
60
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
Linux Kernel
Jun-09, 2016
Linux Kernel
Aug-16, 2016
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
61
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
Linux Kernel
Jun-09, 2016
Linux Kernel
Aug-16, 2016
CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-5829 CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-5829 CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-5829 CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-5829 CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-5829 CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-5829 CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-5829
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
62
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
Linux Kernel
Jun-09, 2016
Linux Kernel
Aug-16, 2016
Linux Kernel
Sep-12, 2016
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
63
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
Linux Kernel
Jun-09, 2016
Linux Kernel
Aug-16, 2016
Linux Kernel
Sep-12, 2016
CVE-2016-6480
CVE-2016-6480
CVE-2016-6480
CVE-2016-6480
CVE-2016-6480
CVE-2016-6480
CVE-2016-6480
CVE-2016-6480
Sample data taken
on Sept-15, 2016
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
Sample data taken
on Sept-15, 2016
In a SUSE data center, today ;-)
64
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
Linux Kernel
Jun-09, 2016
Linux Kernel
Aug-16, 2016
Linux Kernel
Sep-12, 2016
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
In a SUSE data center, today ;-)
65
Linux Kernel
Nov-11, 2015
Linux Kernel
Dec-11, 2015
Linux Kernel
Jan-15, 2016
Linux Kernel
Feb-10, 2016
Linux Kernel
Mar-22, 2016
Linux Kernel
Jun-09, 2016
Linux Kernel
Aug-16, 2016
Linux Kernel
Sep-12, 2016
CVE-2013-7446
CVE-2015-6937
CVE-2015-7872
CVE-2015-7990
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-6480
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0728
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-6480
CVE-2013-7446
CVE-2015-8019
CVE-2015-8539
CVE-2015-8660
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-6480
CVE-2015-8709
CVE-2015-8812
CVE-2015-8816
CVE-2016-0758
CVE-2016-0774
CVE-2016-1583
CVE-2016-2053
CVE-2016-2384
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-6480
CVE-2016-0758
CVE-2016-1583
CVE-2016-2053
CVE-2016-3134
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-6480
CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-4997
CVE-2016-5829
CVE-2016-6480
CVE-2016-6480
CVE-2016-0758
CVE-2016-2053
CVE-2016-4470
CVE-2016-4565
CVE-2016-5829
CVE-2016-6480
Sample data taken
on Sept-15, 2016
December
2015
January
2016February
2016
March
2016
April
2016
May
2016June
2016
July
2016August
2016
September
2016
Key Solution Highlights
66
Available for SLES 12 onwards (x86-64)
Provides fixes for Kernel bugs which affect
Security
Stability
Data Integrity
No runtime performance impact
No interruption of applications while patching
Allows full review of patch source code
Build-in PTF support
Patches available for most recent maintenance
kernels (last 12 months)
Currently based on kGraft OpenSource project
Where does SLE Live Patching make most sense?
... and where not? What‘s your guess?
67
Where does SLE Live Patching make most sense?
... and where not? What‘s your guess?
68
(c) creativecommons.org/licenses/by/3.0
Where does SLE Live Patching make most sense?
... and where not? What‘s your guess?
69
(c) creativecommons.org/licenses/by/3.0
http://cdn.slashgear.com/wp-
content/uploads/2012/10/google-datacenter-tech-21.jpg
Where does SLE Live Patching make most sense?
... and where not? What‘s your guess?
70
(c) creativecommons.org/licenses/by/3.0
http://cdn.slashgear.com/wp-
content/uploads/2012/10/google-datacenter-tech-21.jpg
(c) openSUSE.org
Where does SLE Live Patching make most sense?
... and where not? What‘s your guess?
71
(c) creativecommons.org/licenses/by/3.0
http://cdn.slashgear.com/wp-
content/uploads/2012/10/google-datacenter-tech-21.jpg
(c) openSUSE.org FUJITSU PRIMEQUEST 2800B, (c) Fujitsu
SAP
HANA
Outlook
72
SLE Live
Patching for
ppc64le SLE Live
Patching for
IBM z Systems
User Space
Live PatchingSLE Live
Patching for
Aarch64
Virtualization
Live Patching
Further Information
73
Join SUSE as Live Patching developer
https://jobs.suse.com/job/prague/live-patching-developer/3486/2529381
SUSE Linux Enterprise Live Patching – 60 day Eval
www.suse.com/products/sles-for-sap/
Forrester – Linux vs. Unix Hot Patching – have we reached the tipping point?
http://blogs.forrester.com/richard_fichera/16-05-20-
linux_vs_unix_hot_patching_have_we_reached_the_tipping_point
7-11 November, 2016www.susecon.com
Thank you
74
Hannes Kühnemund
SUSE Product Management
@hakuehnemund
www.linkedin.com/in/hanneskuehnemund
Backup
75
References
One hour of downtime costs $100k for 95% of all enterprises
http://itic-corp.com/blog/2013/07/one-hour-of-downtime-costs-100k-for-95-of-
enterprises/
Kernel Live Patching for ppc64le
http://mpe.github.io/posts/2016/05/23/kernel-live-patching-for-ppc64le/
Forrester – Linux vs. Unix Hot Patching – have we reached the tipping point?
http://blogs.forrester.com/richard_fichera/16-05-20-
linux_vs_unix_hot_patching_have_we_reached_the_tipping_point
Using Live Patching to patch a running SAP HANA system with zero interruption
https://www.youtube.com/watch?v=E9KwTfWeVLg76
