Upload
china-netcloud
View
72
Download
2
Embed Size (px)
Citation preview
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetcloud Training Program
ChinaNetCloud Training
Linux Memory BasicsBy ChinaNetCloudPioneers in OaaS – Operations-as-a-Service
October, 2013
www.ChinaNetCloud.com
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 2
Introduction
● Linux memory is complex and interesting● Class summarizes:
● Types of memory● How it's used● How to troubleshoot memory issues
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 3
Overview, Purpose & Use
● Linux memory is one of the most important areas for engineers to understand
● Used by everything● Often not well-understood● Often the cause of problems
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 4
Introduction
● Memory is simple, in theory● Complex in real use● Plenty of strange things, too● Important to understand● Important to monitor
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 5
Memory Types I
● Physical RAM – Fixed amount, real RAM● Virtual Memory – Virtual, can swap● Shared – Between processes
● Oracle & Postgres use this● Slab – Kernel memory
● /proc/slabinfo & slabtop utility● Includes big RAM users
– tcpmem & inode cache
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 6
Memory Types II
● Page Cache – File system cache● Dirty
● Changed file system data waiting to write to disks● Important for high write systems
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 7
/proc/meminfo
Very useful but complex ●These are items not found by 'free' or 'top'
●MemTotal: Total usable ram (RAM minus kernel binary code)●MemFree: Total free memory, same as free's 'free' output●SwapCached: Memory swapped out, then back in, but still also in the swapfile
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 8
/proc/meminfo
● Active – Recently used, not reclaimed unless necessary● Inactive – Less recently used, likely to be reclaimed ● Dirty – File / Page Cache waiting to be written to disk● Writeback – Memory which is now being written to the disk● Mapped – Memory Mapped files, such as libraries
● Includes Mongo, Varnish, and many others● Slab – Kernel memory, usually 256-512MB
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 9
Memory Page Size
● 4KB page size● Important as many stats are in pages
– Be careful of units ! KB vs. Pages
● sysctl items like tcpmem are mixed● Do not confuse with disk sector size of 0.5KB
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 10
Free -m
● Total – Physical in machine
● Used – by Apps plus Cache, Buffers and maybe Shared
● Free – Totally unused RAM, not important, usually very small
● Shared – by different processes like Oracle / Postgres
● Buffers – Raw block cache to/from disks, not impotrant
● Cached (Page Cache) – Used by disk files cached in RAM
● Swap – Total, Used, Free – Used should be small
● -/+ buffers/cache – Important numbers, inside box is key● PAY ATTENTION to the BOX number – it's all that matters
total used free shared buffers cachedMem: 3961 3901 60 0 121 1232-/+ buffers/cache: 2546 1414Swap: 4683 756 3927
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 11
Top, ps Memory Output
● VIRT – Virtual size● Includes mapped libraries, often very large● Not too useful – IGNORE THIS
● RSS – Resident Set Size● Most important !● Real RAM used by applications, not including swap
● SHR – Shared, such as Oracle / Postgres● Be careful to see processes not threads
● Threads will share RSS, see htop in thread mode
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 12
Swap
● Simple, but not simple● Managed by kswapd● Actual swap used calc
● SwapTotal – SwapCached – SwapFree● Goal is zero swap on servers
● Never let a system actively swap● But some systems will have small swap
● Common to see 100-200MB, more on big system● NUMA defaults can cause swapping
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 13
Problem with Swap
● Swapping FREEZES the swapping process ● So, a 32MB byte swap of MySQL RAM
● Freezes ALL of MySQL for many seconds● Very bad for applications
● Bad for any single process system● MySQL, Nginx, HAProxy, etc. (not Apache)
● Also uses valuable IO, slowing DBs, etc.● Goal is to NEVER swap real applications
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 14
Swappiness
● Tells kernel which is more important● File system Cache vs. App Memory
● Default is 60 – Means cache more important● Stupid for servers – will swap even with free RAM
● Set to 1, always (used to be 0, now use 1)● Won't swap until all RAM used by applications● Note some swap anyway (see next slide)
● Set by sysctl● Check in /proc/sys/vm/swappiness
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 15
Swap still happens
● Linux likes to swap a little ● Even with free RAM and swappiness = 0● Some kernel data wants to swap● Can pre-swap SwapCached data● Still a mystery, but it's okay
● Very large systems (64GB+ might swap 1-2GB)● Turning off can cause kswapd to go crazy● Watch vmstat swapin/out, make sure very small
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 16
NUMA - Can cause swap
● NUMA – Non-Uniform Memory Access● Each CPU has its own RAM● Slower to use 'other' CPU's RAM● Standard now on all Intel servers
● Causes RAM inbalance on big RAM processes● Like MySQL, MongoDB, Java
● Install & use numa-utils package, numactl app● Set all big processes to full interleave:
● numactrl –interleave all mysqld . . .
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 17
Dirty Memory & Ratios
● Causes strange problems & performance issues● See /proc/meminfo
● Dirty is the changed FileSystem cache data● Must be written to disk● Can cause big problems if too high● Two ratios control
● /proc/sys/vm/dirty_background_ratio / _bytes● /proc/sys/vm/dirty_ratio / _bytes
● Freezes all writing processes when dirty > dirty_ratio● Calculation not clear, but keep dirty data < 20% of RAM
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 18
OOM Killer
● Kills process when no more RAM or Swap● Chooses process to die via a score
● Biggest RAM user (MySQL) often dies first● See /proc/<pid>/oom_score
● Always a message in dmesg - /var/log/kernel● Can adjust score to control (advanced)● Best practice to monitor swap and log
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2013 ChinaNetCloud 19
Summary
● Memory important to understand● Especially basics & definitions● Understand swap & swappiness● Watch NUMA on big systems● Memory is cheap – buy more !
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
Copyright 2015 ChinaNetCloud 20
About ChinaNetCloud
ChinaNetCloudRunning the World's Internet Servers 管理全球服务器
ChinaNetCloud [email protected]
www.ChinaNetCloud.com
Beijing Office:
Lee World Business Building #305
57 Happiness Village Road,
Chaoyang District
Beijing, 100027 China
Silicon Valley Office:
California Avenue
Palo Alto, 94123 USA
Shanghai Headquarters:
X2 Space 1-601, 1238 Xietu Lu
Shanghai, 200032 China
T: +86-21-6422-1946 F: +86-21-
6422-4911