Red Hat Performance Engineering

RHEL 7 Low Latency Update

Joe MarioOct 26, 2016Senior Principal EngineerRed Hat Performance Engineering

Red Hat Performance Engineering

Low Latency Performance Tuning Guide for Red Hat Enterprise Linux 7

• Tactical tuning overview for latency-sensitive workloads.• Emphasizes impactful new features included in RHEL7:

• CPU/power management• NUMA• tuned profiles• scheduling• network tunables• kernel timers.• "de-jittering" CPU cores• tracing techniques https://access.redhat.com/articles/

1323793

Red Hat Performance Engineering

throughput-performance (R7 default)• governor=performance• energy_perf_bias=performance• min_perf_pct=100• readahead=4096• kernel.sched_min_granularity_ns = 10000000• kernel.sched_wakeup_granularity_ns = 15000000• vm.dirty_background_ratio = 10• vm.swappiness=10

Red Hat Performance Engineering

Children/Grandchildren

Tuned network-latency Profilelatency-performance

force_latency=1governor=performanceenergy_perf_bias=performancemin_perf_pct=100kernel.sched_min_granularity_ns=10000000vm.dirty_ratio=10vm.dirty_background_ratio=3vm.swappiness=10kernel.sched_migration_cost_ns=5000000

network-latencyinclude=latency-performancetransparent_hugepages=nevernet.core.busy_read=50net.core.busy_poll=50 net.ipv4.tcp_fastopen=3kernel.numa_balancing=0

Children

Parentslatency-

performancethroughput-performance

network-latencynetwork-

throughput

virtual-host

virtual-guest

balanced

desktop

GatewayFeed HandlerMatching

Eng.

Red Hat Performance Engineering

Performance Optimizations in RHEL7Networking

• Full support for PTP1588v2• Route cache → F.I.B.• irqbalance handles NUMA• busy_poll, tcp_fastopen• nohz_full (tickless while active)• Byte Queue Limits• TCP Small Queues

CPU• Support for all new CPUs• AVX2 instruction support• RHEL-RT • RHEL-RT/KVM (tech preview in 7.3)

Power Management

• intel_pstate

• tuned does most heavy lifting

Memory

• Automatic NUMA Balancing

• Tunable workqueues (writeback)

Red Hat Performance Engineering

RHEL7 BUSY_POLL Socket Option• Socket-layer code

polls receive queue of NIC

• Replaces interrupts and NAPI

• Retains full capabilities of kernel network stack

Higher is better

Red Hat Performance Engineering

Accelerating Red Hat Enterprise Linux 7-based Linux Containers with Solarflare OpenOnload• Config and tuning guidance for OpenOnload-accelerated Linux

containers• Emphasizes impactful new features included in RHEL7:

• Docker• Atomic• Super-privileged Containers• Low Latency Tuning• tuned profiles

https://access.redhat.com/articles/1407003

Red Hat Performance Engineering

RHEL 7.1 + OpenOnloadBare Metal/Containers/SR-IOV

Red Hat Performance Engineering

Full DynTicks PatchsetPatchset Goal:● Stop interrupting userspace

tasks● Move timekeeping to non-

latency-sensitive cores● If nr_running=1, then

scheduler/tick can avoid that core

● Default disabled...Opt-in via nohz_full cmdline option

Kernel Tick:● timekeeping

(gettimeofday)● Scheduler load balancing● Memory statistics

(vmstat)

Red Hat Performance Engineering

RHEL6 and 7 Tickless

Tick No No No Tick

Userspace Task Timer Interrupt Idle

Time (CONFIG_HZ=1000)

Red Hat Performance Engineering

RHEL7 nohz_full

Userspace Task Timer Interrupt Idle

Time (CONFIG_HZ=1000)

No Ticks

Red Hat Performance Engineering

NFV/Realtime Performance

Red Hat Performance Engineering

RT, KVM-RT/NFV Tuned Profiles

Red Hat Performance Engineering

Realtime Scheduler Latency Jitter Plot(rt-kvm is tech preview in RHEL 7.3)

Red Hat Performance Engineering

Scheduler Latency (cyclictest)

Red Hat Performance Engineering

RHEL 7.x Network Performance Intel Haswell EP, 12-40Gb ports (6 cards)

Red Hat Performance Engineering

NFV 40G Packets/Sec DPDK (64 byte UDP)

Red Hat Performance Engineering

40G Network Data/Tuned Networks

Red Hat Performance Engineering

Not quite in RHEL yet (hopefully 7.4) Intel Cache Allocation Technology

© 2016 Intel Corporation.

Red Hat Performance Engineering

ReferencesLow Latency Tuning Guide for Red Hat Enterprise Linux 7https://access.redhat.com/articles/1323793

Accelerating Red Hat Enterprise Linux 7-based Linux Containers with Solarflare OpenOnloadhttps://access.redhat.com/articles/1407003

How do I create my own tuned profile on RHEL7 ?https://access.redhat.com/solutions/731473