Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Deterministic Container Resource Management inDerivative Clouds
Chandra Prakash, Prashanth, Umesh Bellur, Purushottam Kulkarni
Department of Computer Science and EngineeringIndian Institute of Technology Bombay
19th April, 2018
International Conference on Cloud Engineering
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 1 / 26
Derivative cloud
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 2 / 26
Dual control over resources
I Hypervisor and guest OS both control the same resourcesI Hypervisor not aware of containers requirements
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 3 / 26
Hypervisor mechanism under consideration
I Ballooning is used to achieve memory overcommitment1
I vCPU scaling is used to reduce scheduling overheads in overcommitted situation 2
1C. A. Waldspurger, "Memory resource management in vmware esx server," ACM SIGOPS Operating Systems Review, vol.
36, no. SI, pp. 181-194, 2002.2
L. Cheng, J. Rao, and F. Lau, "vscale: automatic and efficient processor scaling for smp virtual machines," in Proceedingsof the 11th European Conference on Computer Systems, ACM, 2016.
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 4 / 26
Undesirable effects due to ballooning
Happens Desired
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 5 / 26
Existing memory reclamation in containers
I Memory provisioning knobs: Hard-Limit and Soft-LimitI exceed: difference between memory usage and Soft-LimitI SMR (Soft Memory Reclaimed): memory reclaimed from local LRUI GLR (Global LRU Reclaimed): memory reclaimed from global LRU
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 6 / 26
Existing memory reclamation in containers
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 7 / 26
Impact of ballooning
Set-up
I Memory reclamation rate: 2 GB every 30 seconds (generatedfrom host after 100 seconds)
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 8 / 26
Impact of ballooning
Default configuration of four containersContainer Hard-limit(GB) Soft-limit(GB) Key size (# records)Redis-Low 2 0.5 500KRedis-High 4 1 1000KMongo-Low 2 0.5 500KMongo-High 4 1 1000K
0
0.5
1
1.5
2
2.5
3
3.5
0 50 100 150 200 250 300 350 400
Memory usage ratio
Time (sec)
MongoDB ratio
Redis ratio
Desired ratio
Memory usage ratio
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 9 / 26
Impact of ballooning
I Existing knobs (limits) do not guarantee proportionatememory allocation during memory pressure situations
0
0.5
1
1.5
2
2.5
3
3.5
0 50 100 150 200 250 300 350 400
Memory usage ratio
Time (sec)
MongoDB ratio
Redis ratio
Desired ratio
Memory usage ratio
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 10 / 26
CPU provisioning Issues
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 11 / 26
Impact of vCPU scaling
Experimental setupVM configuration 7 vCPUs and 8GB Memory
Number of containers inside VM 3
CPU allocation ratio 1:1:4
Benchmark Sysbench
vCPU scaling down frequency 1 vCPU every 120s (vCPU1,2,3,&4)
vCPU mapping using cpuset.cpus C1: vCPU1, C2: vCPU2, C3: vCPU3,4,5,&6
0
100
200
300
400
1-120 121-240 241-360 361-480 481-600
Avg. %CPU utilization
Time Range(S)
Container1
Container2
Container3
CPU utilization without pinning
0
100
200
300
400
1-120 121-240 241-360 361-480 481-600
Avg. %CPU utilization
Time Range(S)
Container1
Container2
Container3
CPU utilization with pinning
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 12 / 26
Impact of vCPU scaling
I Pinning and scaling =⇒ non-deterministic CPU utilizationI Desired goal: achieve pinning benefits + maintain CPU share
0
100
200
300
400
1-120 121-240 241-360 361-480 481-600
Avg. %CPU utilization
Time Range(S)
Container1
Container2
Container3
CPU utilization without pinning
0
100
200
300
400
1-120 121-240 241-360 361-480 481-600
Avg. %CPU utilization
Time Range(S)
Container1
Container2
Container3
CPU utilization with pinning
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 13 / 26
Summary of issues in nesting setup
I Ballooning may fail to satisfy container requirements
I vCPU scaling may not respect cpu share with cpu pinning
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 14 / 26
Our approach
I Native cloud provider can be public or privateI We can’t control or change hypervisor in case of public cloudI We provide solution at guest OS level
Default approach Our approach
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 15 / 26
Proposed memory policies
Proportionate memory allocationI Allocate memory according to credit share of containers
Application-specific differentiated memory reclamationI Protect memory sensitive container(s) from memory reclamation
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 16 / 26
Proposed CPU policies
Maximize dedicated vCPU while maintaining allocation ratioI To get maximum benefits of pinning
Provide pinned vCPU(s) to a subset of containersI Based on application nature or user requirement
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 17 / 26
Contribution
I Modified the memory reclamation logic in memory cgroupsubsystem
I Provided an additional definition of exceed
I Performed several modifications in Linux kernel» Added extra parameters in memory and cpu cgroups» Added control to maximize SMR» Provided knob to control reclamation chunk size
I Created a cpuset calculator in user space
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 18 / 26
Modified memory reclamation
» exceed = memory_usage – proportionate_share
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 19 / 26
Effectiveness of memory policies
I Ratio of memory weights: 1:2
0
0.5
1
1.5
2
2.5
3
3.5
0 50 100 150 200 250 300 350 400
MongoDB ratioRedis ratio
Desired ratio
Time (sec)
Mem
ory
usa
ge r
atio
pressureNo memory
Memory pressure exceed of all ≰ 0
Memory pressure exceed of all ≤ 0
Memory usage ratio (default)
0
0.5
1
1.5
2
2.5
3
0 50 100 150 200 250 300 350 400
MongoDB ratioRedis ratio
Desired ratio
Time (sec)M
emo
ry u
sage
rat
io
Memory usage ratio (with control)
I Able to maintain memory usage ratio when exceed of allcontainers become less than or equal zero (after 300 second)
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 20 / 26
Application specific reclamation
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500 600
Memory usage (MB)
Time (sec)
TPC-C: default
TPC-C: with control
YCSB: default
YCSB: with control
Free Memory
Memory usage
0
200
400
600
800
1000
1200
1400
1600
1800
0 100 200 300 400 500 600
Throughput (op/sec)
Time (sec)
YCSB: default
YCSB: with control
YCSB throughput
I Memory is not reclaimed from YCSB application container(memory sensitive) and it’s throughput remains intact
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 21 / 26
vCPU reallocation design
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 22 / 26
Effectiveness of vCPU reallocationExperimental setup
VM configuration 7 vCPUs and 8GB Memory
Number of containers inside VM 3
CPU allocation ratio 1:4:5
Benchmark Sysbench & Twitter
VCPU scaling down frequency 1 vCPU every 120s (vCPU1,2,3,& 4)
0
100
200
300
400
1-120 121-240 241-360 361-480 481-600
Avg. %CPU utilization
Time Range(S)
Container1
Container2
Container3
CPU utilization by each container
0
500
1000
1500
2000
2500
3000
3500
0 100 200 300 400 500 600Throughput (op/sec)
Time (sec)
With pinningWithout pinning
Twitter throughput with scaling down vCPUs
I Able to maintain CPU share along with pinning
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 23 / 26
Conclusion & future work
Conclusion:I Quantified the impact of hypervisor actions on containers running
inside VMI Proposed user-defined policies to mitigate the impact of
hypervisor actionsI Demonstrated the effectiveness of memory and CPU policies
empiricallyFuture work:
I Design an efficient algorithm for container placement in derivative(nested) setup
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 24 / 26
Thank youQuestions???
Email id: [email protected]
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 25 / 26
Modifications in Linux kernel
I Added a weight parameter in memory cgroup and a pinparameter in cpu cgroup.
I Modified the balance_pgdat() routine (Linux kernel version 4.7).Listing 1 : Original reclamation logic
For every rec lamat ion request :SMR( ) ;GLR ( ) ;
Listing 2 : Modified reclamation logic
For every rec lamat ion request :NoOfReclaimedPages = SMR( ) ;
i f ( NoOfReclaimedPages ==0) :GLR ( ) ;
I Created a kernel module to control the reclamation chunk size.
Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 26 / 26