Upload
salman-baset
View
90
Download
1
Embed Size (px)
Citation preview
A Survey of Container Security in 2016: A Security Update on Container PlatformsSalman A. Baset, Research Staff Member, @salman_basetPhil Estes, Senior Technical Staff Member, @estesp
Recap and Background
2015• Unraveling Docker Security: Lessons from Production
Cloud (OpenStack Summit Tokyo, October 2015)
2016• Container Security (Docker London, June 2016)• How Secure is Your Container? (ContainerCon Berlin,
August 2016)
Today’s talk:• Container Security: Survey and Overview, and a
developer focused comparison of secure application deployment & configuration (OpenStack Summit Barcelona, October 2016)
Outline• Container Overview• Usage and Threat Models• Container Security Evolution• Developer Focus: Running Applications (Apache) Inside
Containers vs Host• Why Containers are Better for Security?• What are the New Security Issues?
• Docker vs. LXC vs. Rocket (NCC group report)• Ongoing work in Container Security
Raise your hand if you think that…• Running applications in hosts (VM / baremetal) vs. running
them inside containers on a VM / baremetal is: • more “secure”• no difference• less “secure”
• We think that running applications inside a container is more “secure” than running them directly on a VM / baremetal
• This is because containers allow deep visibility, and usable and developer-focused security.
But first what is a container?
What makes a container?
pid mount
IPC
user network
uts
> Linux kernel namespaces provide the isolation (hence “container”) in which we place one or more processes
> Linux kernel cgroups (“Control groups”) provide resource limiting and accounting (CPU, memory, I/O bandwidth, etc.)
Container usage and threats?
Single tenant, trusted codeLess concern regarding
malicious attack vectorsMulti-tenant, (organizationally)
trusted code ”Noisy neighbor” a higher priority
than protection from unknownsMulti-tenant, untrusted code
Similar to internet-facing application, but with addition of potential malicious “neighbors”
DoS Host (use up CPU, memory, disk), Forkbomb
Container, even with isolation, is able to use up host OS resources
Access host/private information
Container is allowed to access (maybe inadvertently) host details
Kernel modification/insert module
Given the shared kernel underlying containers; modifying the kernel’s configuration/code/inserting code
Administrative access (API socket access)
Access to the container “control” socket/system, allowing full admin privileges of host system
Exploits in OS libraries/code in container
Latent/unpatched exploits in libraries and/or binaries in the container’s filesystem
Container provenance
Validating/certifying the container image I am running is what I expect
THREAT DESCRIPTION
Kernel and Container Security Evolution• 2015
• Content Addressability (image spec, phase 1) – Docker 1.6• Default ulimits for all containers – Docker 1.6• Docker Content Trust – (notary) image provenance/signing – Docker 1.8
• 2016• Full migration to content addressability for images/layers – Docker 1.10• User namespaces – Docker 1.10• Secure computing (libseccomp) – Docker 1.10• --pids-limit (cgroups pid limitation) – Docker 1.11 (kernel 4.3 +)• cgroups “v2” – (kernel 4.5+)• --no-new-privileges (limit process escalation) – Docker 1.11• Storage driver quotas (limited) – Docker 1.12• Secure by default multi-node orchestration (mutual TLS) – Docker 1.12
Kernel and – save chart until we decide-2015 2016
Kernel – 3.13 Docker – 1.6 Kernel – 4.4 Docker – 1.12
cgroup pid limits
Configuring for Security: Hard & Tedious Work!
• Center for Internet Security (CIS) security “benchmarks”. • Consensus driven approach for secure configuration for host and key applications.• https://www.cisecurity.org/
• Operating Systems• Red Hat 7 CIS benchmark – v2.1.0 – June 2nd, 2016
• 304 pages• Ubuntu 14.04 CIS benchmark – v2.0.0 – Sep 30, 2016
• 296 pages
• Apache HTTP Server 2.4 benchmark – v1.3.0 – May 31st, 2016• 163 pages
• Docker 1.12 CIS benchmark – v1.0.0 – August 15, 2016• 189 pages
• OWASP rules for Apache• 60 mod security rules
Installing Applications in a Container vs. Directly on Host• Host configuration and hardening is needed whether
applications run in a container on a host, or directly on host. This is typically a one-time operation.
• Launching applications in a container on a host [is] a recurring operation. Container makes it easy to secure host from applications, making security usable.
HostHost
Container
• Apache webserver is among the most popular web servers in use today.
• We think that deploying Apache web server in a container is more secure than deploying directly on a host.
• Let’s compare Apache installation in a container on a host versus directly on a host to better understand our assertion.
Running Apache web server on a container vs. host (1/10)
Filesystem Size Mounted on CIS rules/dev/disk1 100G /
/dev/disk1 10G /tmp Ubuntu CIS 1.1.2
/dev/disk2 100G /var/log Ubuntu CIS 1.1.10
/dev/disk3 100G /var/log/audit Ubuntu CIS 1.1.11
/dev/disk4 100G /mnt/httpd
HOST
STEP 1: • Configuring host file system• Configure separate partitions for Docker• Install Docker Engine
Preparing host list is not an exhaustive list and is meant as an illustration.
Filesystem Size Mounted on CIS rules
/dev/disk1 100G /
/dev/disk1 10G /tmp Ubuntu CIS 1.1.2
/dev/disk2 100G /var/log Ubuntu CIS 1.1.10
/dev/disk3 100G /var/log/audit Ubuntu CIS 1.1.11
/dev/disk4 100G /var/lib/docker Docker CIS 1.1
/dev/disk5 100G /var/dockervols
CONTAINER (DOCKER) HOST
Agents / daemonsSSHDockerRunc
Agents / daemonsSSH
STEP 1: • Configuring host file system• Configure separate partitions for Apache
Goal: Configure separate partition
Running Apache web server on a container vs. host (2/10)
Configure Config file Description CIS rulesSSH /etc/ssh/sshd_config Configure SSH
properlyUbuntu CIS 5.2
User accounts
/etc/passwd, /etc/shadow
Configure user accounts per strict policy
Ubuntu CIS 5.4
Apache user Configure Apache user and run apache under it
Apache CIS 1.3
Configure Config file Description CIS rulesSSH /etc/ssh/sshd_config Configure SSH
properlyUbuntu CIS 5.2
User accounts
/etc/passwd, /etc/shadow
Configure user accounts per strict policy
Ubuntu CIS 5.4
User namespace user
Configure Docker engine with user namespace
Docker CIS 2.8
Apache Docker Image
Prepare Docker image for Apache with Apache user.
Apache CIS 1.3
HOST
STEP 2: • Manually configure Docker engine with user
namespace if non-root user not defined in Docker image.
• Use of kernel namespaces is by default.
CONTAINER (DOCKER) HOST
Agents / daemonsSSHDockerRunc
Agents / daemonsSSH
STEP 2: • Configure non-root user for Apache.• Manually configure kernel namespaces.
Goal: Run processes as non-root utilizing namespaces. Essential for deprivileging
Running Apache web server on a container vs. host (3/10)
Name Configuration files
Description CIS rules
Audit daemon
/etc/audit/auditd.conf
Record system call activity Ubuntu CIS 4.1
Rsyslog /etc/rsyslog.conf A daemon for sending logs to a central server
Ubuntu CIS 4.2
Logging and log rotation
Configure log rotation to avoid disk full.
Ubuntu CIS 4.3Apache CIS 1.6
Monitoring Configure monitoring Apache CIS 1.6
Name Configuration files
Description CIS rules
Audit daemon
/etc/audit/auditd.conf
Record system call activity
Ubuntu CIS 4.1Docker CIS 1.7-1.15
Rsyslog /etc/rsyslog.conf A daemon for sending logs to a central server
Ubuntu CIS 4.2
Logging and log rotation
Configure log rotation to avoid disk full.
Ubuntu CIS 4.3Docker CIS 2.12
Monitoring Configure monitoring of containers
Docker CIS 6.2
HOST
STEP 3: • Configure audit, log collection, rotation, monitoring• Configure audit for Docker Engine• Collect Docker Engine and container logs
CONTAINER (DOCKER) HOST
Agents / daemons1. SSH2. Docker3. Runc4. Audit5. Rsyslog6. Monitoring
STEP 3: • Configure audit, log collection, rotation, monitoring• Collect Apache logs
Agents / daemons1. SSH2. Audit3. Rsyslog4. Monitoring
Goal: Track system call activity, logs, and monitoring
Running Apache web server on a container vs. host (4/10)
Configure Config file Description CIS rules
NTP /etc/ntp.conf Ubuntu CIS 2.2.1
DNS DNS server not enabled. Ubuntu CIS 2.2.8
/proc/sys/net /etc/sysctl.conf /proc/net
Network related configs, e.g., net.ipv4.ip_forward=0
Ubuntu CIS 3.1/3.2
Firewalls iptables / security groups
Configure host firewalls Ubuntu CIS 3.6
Configure Config file Description CIS rules
NTP /etc/ntp.conf Ubuntu CIS 2.2.1
DNS DNS server not enabled. Ubuntu CIS 2.2.8
/proc/sys/net
docker
/etc/sysctl.conf /proc/net/etc/docker
Network related configs, e.g., net.ipv4.ip_forward=1--sysctl net.ipv4.ip_forward=0
Ubuntu CIS 3.1/3.2
Firewalls iptables / security groups
Configure host firewalls Ubuntu CIS 3.6
HOST
STEP 4: • Configure network and related services• Disable IP forwarding when starting container
engine.
CONTAINER (DOCKER) HOST
Agents / daemons1. SSH2. Docker3. Runc4. Audit5. Rsyslog6. Monitoring
STEP 4: • Configure network and related services
7. NTP8. DNS9. /etc/default/docker10. Firewalls
Agents / daemons1. SSH2. Audit3. Rsyslog4. Monitoring5. NTP6. DNS
7. /proc/sys/net8. Firewalls
Goal: Configure network services, firewalls, IP forwarding
Running Apache web server on a container vs. host (5/10)
Name Configuration files Description CIS rulesSystem updates
/etc/apt/sources.list.d/*
Configure system updates using an agent
Ubuntu CIS 1.8
Antivirus / Malware detector
A daemon for detecting any malware on host
Name Configuration files Description CIS rulesSystem updates
/etc/apt/sources.list.d/*
Configure system updates using an agent
Ubuntu CIS 1.8
Antivirus / Malware detector
A daemon for detecting any malware on host
HOST
STEP 5: • Configure patching, and malware / antivirus agents
on host
CONTAINER (DOCKER) HOST
Agents / daemons1. SSH2. Docker3. Runc4. Audit5. Rsyslog6. Monitoring
STEP 5: • Configure patching, and malware / antivirus agents
on host
7. NTP8. DNS9. /etc/default/docker10. Firewalls 11. Patching agent12. Malware agent
Agents / daemons1. SSH2. Audit3. Rsyslog4. Monitoring5. NTP6. DNS
7. /proc/sys/net8. Firewalls9. Patching agent10. Malware agent
Goal: Configure software updates and malware / antivirus
Running Apache web server on a container vs. host (6/10)
Name Configuration files
Description CIS rules
Configure AppArmor (MAC)
Configure Apache AppArmor profile
/etc/apparmod.d/* Configure apparmor daemon and all profiles in enforcing
Ubuntu CIS 1.6
Apache CIS 1.12
Name Configuration files
Description CIS rules
Configure AppArmor (MAC)
Do not disable AppArmor
/etc/apparmod.d/* Configure apparmor daemon and all profiles in enforcing
Ubuntu CIS 1.6
Docker CIS 5.1
HOST
STEP 6: • Do not disable default Docker AppArmor profile.
Applied to running containers.
CONTAINER (DOCKER) HOST
Agents / daemons1. SSH2. Docker3. Runc4. Audit5. Rsyslog6. Monitoring
STEP 6: • Create an AppArmor profile for Apache. Complicated!
7. NTP8. DNS9. /etc/default/docker10. Firewalls 11. Patching agent12. Malware agent13. AppArmor
Agents / daemons1. SSH2. Audit3. Rsyslog4. Monitoring5. NTP6. DNS
7. /proc/sys/net8. Firewalls9. Patching agent10. Malware agent11. AppArmor
Goal: Configure mandatory access control for defense in depth
Running Apache web server on a container vs. host (7/10)
Denial of service Description / mitigation CIS rulesNetwork Configure Apache for network DoS
mitigation.Apache CIS 1.9
Memory systemctl MemoryLimit=500M …
Resources (e.g., fork)
CPU systemctl CPUShares=…
Disks systemctl IO[Read|Write]BandwidthMax=…
ulimits Set ulimits
Denial of service Description / Mitigation CIS rulesNetwork Open issue #26767 (WIP)
Memory docker run -m
Resources (e.g., fork)
docker run --pids-limit
CPU docker run --cpu-set-cpus, --cpu-shares
Disks docker run --device-[read|write]-[i|b]ps
ulimits Set ulimits for open file descriptors Docker CIS 5.18
HOST
STEP 7: • Allow resource limits with an easy to use interface
to cgroups and ulimits for ANY container.
CONTAINER (DOCKER) HOST
Agents / daemons1. SSH2. Docker3. Runc4. Audit5. Rsyslog6. Monitoring
STEP 7: • Use systemd / upstart or manually configure
resource control using cgroups and ulimits using.Cumbersome.
7. NTP8. DNS9. /etc/default/docker10. Firewalls 11. Patching agent12. Malware agent13. AppArmor
Agents / daemons1. SSH2. Audit3. Rsyslog4. Monitoring5. NTP6. DNS
7. /proc/sys/net8. Firewalls9. Patching agent10. Malware agent11. AppArmor
Goal: Prevent Apache from DoSing host or other containers
Kernel: 4.5+ “cgroups v2”
Running Apache web server on a container vs. host (8/10)
Capabilities Description / mitigation CIS rules
Various Manual configuration
Capabilities Description / Mitigation CIS rules
Various Load a module, mount, network admin. Docker starts container with limited capabilities.
Docker CIS 5.3
HOST
STEP 8: • Launch containers as non-root with limited Linux
capabilities. Defaults are good.
CONTAINER (DOCKER) HOST
Agents / daemons1. SSH2. Docker3. Runc4. Audit5. Rsyslog6. Monitoring
STEP 8: • Manual selection and setting of capabilities is
cumbersome.
7. NTP8. DNS9. /etc/default/docker10. Firewalls 11. Patching agent12. Malware agent13. AppArmor
Agents / daemons1. SSH2. Audit3. Rsyslog4. Monitoring5. NTP6. DNS
7. /proc/sys/net8. Firewalls9. Patching agent10. Malware agent11. AppArmor
Goal: Further deprivilege running process and containers through limited Linux capabilities.
Although Apache is launched by a non-root user in container as well as on host, as part of defense in depth, it is good practice to further reducing capabilities available to the user.
Running Apache web server on a container vs. host (9/10)
Descriptiom CIS rules
Manual configuration of seccomp
Preparing host list is not an exhaustive list and is meant as an illustration.
Description CIS rules
Do not disable default seccomp profile Docker CIS 5.12
HOST
STEP 9: • Launch containers with limited set of allowed Linux
system calls. Defaults are good.
CONTAINER (DOCKER) HOST
Agents / daemons1. SSH2. Docker3. Runc4. Audit5. Rsyslog6. Monitoring
STEP 9: • Manually configuring undesired system calls
is cumbersome.
7. NTP8. DNS9. /etc/default/docker 10. Firewalls 11. Patching agent12. Malware agent13. AppArmor
Agents / daemons1. SSH2. Audit3. Rsyslog4. Monitoring5. NTP6. DNS
7. /proc/sys/net8. Firewalls9. Patching agent10. Malware agent11. AppArmor
Goal: Further deprivilege running containers by limiting Linux system calls (Seccomp).
Although Apache is launched by a non-root user in container as well as on host with limited capabilties, as part of defense in depth, it is good practice to further reduce system calls that can be invoked.
Running Apache web server on a container vs. host (10/10)
STEP 10: • Configure Apache in a container. • Configuring Apache firewalls is a breeze.
Agents / daemons1. SSH2. Docker3. Runc4. Audit5. Rsyslog6. Monitoring
STEP 10: • Configure Apache on host.• Potential cumbersome iptables configuration.
7. NTP8. DNS9. /etc/default/docker 10. Firewalls 11. Patching agent12. Malware agent13. AppArmor14. Apache
Agents / daemons1. SSH2. Audit3. Rsyslog4. Monitoring5. NTP6. DNS
7. /proc/sys/net8. Firewalls9. Patching agent10. Malware agent11. AppArmor 12. Apache
Goal: Configure Apache for optimum security.
Apache config description CIS rules
Minimize Apache modules Apache CIS 1.2
Run Apache as a non-root user even if user name space is used
Apache CIS 1.3
Apache Access Control Apache CIS 1.4
Minimize features, content, and options Apache CIS 1.5
Configure TLS Apache CIS 1.7
Configure limits Apache CIS 1.8
HOSTCONTAINER (DOCKER) HOST
Not an exhaustive list and is meant as an illustration.
Apache config description CIS rules
Minimize Apache modules Apache CIS 1.2
Run Apache as a non-root user. Apache CIS 1.3
Apache Access Control Apache CIS 1.4
Minimize features, content, and options Apache CIS 1.5
Configure TLS Apache CIS 1.7
Configure limits Apache CIS 1.8
Running Apache web server on a container vs. host• A host is isolated from a misbehaving Apache running inside a
container. Various agents running on host are likely to remain functional, allowing root-cause analysis later.
• Same mechanisms are used to isolate one container from another on the same host.
HostHost
Container
Agents / daemons1. SSH2. Docker3. Runc4. Audit5. Rsyslog6. Monitoring7. NTP8. DNS9. /etc/default/docker /proc is mounted read only for unprivileged containers 10. Firewalls 11. Patching agent12. Malware agent13. AppArmor14. Apache
Agents / daemons1. SSH
2. Audit3. Rsyslog4. Monitoring5. NTP6. DNS7. /proc/sys/net
8. Firewalls 9. Patching agent10. Malware agent11. AppArmor12. Apache
Running Application in a Container vs. Host – Defense in Depth
Configure partitions
Run as non-root, utilizing user and kernel namespaces
Configure log, monitoring, audit
Configure network, related services, disable IP forwarding
Configure patching and anti-virus agents
Configure mandatory access control
Prevent application from DoSing a host
Further deprivilege application through subset of Linux capabilities
Further Deprivilege application by limiting allowed system calls
Configure application security
HostContainer
Host
Manual
Manual
Manualper app
One time(in Docker)
Manualper app
ManualOne time
Step 1 Step 2 Step 3
Manual
Manual
Step 4
Manual
Manual
Step 5 Step 6
Manualper app
Out of box(in Docker)
Out of box(in Docker)
Step 7
Manualper app
Step 8
Manualper app
Out of box(in Docker)
Manualper app
Out of box(in Docker)
Step 9 Step 10
Manualper app
Manual
What are the New Security Issues When Running Applications Inside Containers?
• Running containers as non-root• If a Linux user is not setup inside Dockerfile or Appc file to use the
application, it is best to enable user name spaces in Docker Engine.• We recommend to always enable user namespaces in Docker Engine.
• While using Docker, IP forwarding for containers should be disabled in Docker Engine.
• Docker / Rocket API is powerful. Enable with caution.
Observations from Production
• Application security misconfiguration is among the top causes for container compromise.
• IBM Vulnerability Advisor can identify vulnerable application packages and application security misconfigurations for containers in IBM Containers platform.
Docker vs LXC vs rkt
• Users/packagers won’t turn on security if it’s difficult (AppArmor profiles are hard to write; SELinux can be even harder)
• Sane defaults are tricky as well - someone’s app won’t work and they will complain
• Docker painstakingly tries to find a balance (e.g. DCT off by default, allowance for insecure registries)
* NCC Group report “Understanding and Hardening Linux Containers”, v1.1, p. 97, section 9.13
Ongoing work in Container Security• Not yet namespaced
• /dev, /proc/sys/*• However /proc is mounted as read-only
• Container security work == Kernel security work• Containers share kernel and make use of kernel isolation features
• Use hypervisor isolation for containers• runC is a step in this direction• Replace it with a container run-time that leverages hypervisor isolation,
e.g., Intel Clear Containers• Unikernels• Fully unprivileged containers
Thank You