Upload
lamdang
View
215
Download
0
Embed Size (px)
Citation preview
D-Storm: Dynamic Resource-Efficient Scheduling ofStream Processing Applications
Xunyun Liu and Rajkumar Buyya
Cloud Computing and Distributed Systems (CLOUDS) LaboratorySchool of Computing and Information Systems
The University of Melbourne, Australia
December, 2017
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 1 / 31
Outline
1 BackgroundStream ProcessingApache StormIssues with the Existing Scheduler
2 Solution OverviewOur ApproachFramework Overview
3 Dynamic Resource-Efficient SchedulingProblem FormulationScheduling Algorithm
4 Evaluation
5 Related Work
6 Conclusions and Future Work
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 2 / 31
Outline
1 BackgroundStream ProcessingApache StormIssues with the Existing Scheduler
2 Solution OverviewOur ApproachFramework Overview
3 Dynamic Resource-Efficient SchedulingProblem FormulationScheduling Algorithm
4 Evaluation
5 Related Work
6 Conclusions and Future Work
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 3 / 31
Stream Processing
Stream Data
Arrive continuously in real time, possible infinite
Various data sources & various data structures
Transient value & short data lifespan
Asynchronous & unpredictable
Volume(Terabytes, Petabytes)
Velocity(Real-time, Low latency)
Variety(Social Networks, logs,
Sensor data)
Stream Data(Supported by Data Stream
Management System)
Batch Data(Supported by
Hadoop, DBMS)
Table: Example usages of stream data versus batch data
Batch Data Stream Data
Log Analysis What happened an hour ago? What is happening in the system?Billing Analysis How much did a user spend
in the last billing period?Notify a user that the billing limitis approaching
Fraud Prevention Audit / forensic evidence Intervene to stop the ongoing fraud
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 4 / 31
Stream Processing
Stream Processing
Stream processing is a technique that allows for the collection, integration,analysis, visualization, and system integration of stream data in real time,powering on-the-fly analytic that leads to immediately actionable insights.
Process-once-arrival
Queries over recent data on rolling windowGenerally independent computationsIncremental update of results
Queries run continuously unless beingexplicitly terminated
Suitable for latency-sensitive scenarios
Stream Data
Continuous
Queries
Stream Data
Query
Results
Stream Processing
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 5 / 31
Stream Processing
Logic level
Inter-connected operatorsData streams flow throughthese operators to undergodifferent types of computation
Middleware level
Data Stream ManagementSystem (DSMS)Apache Storm, Samza ...
Infrastructure level
A set of distributed hosts incloud or cluster environmentOrganised in Master/Slavemodel
Distributed Stream Processing System
Split Op Op Op
OpJoin &
SplitOp Join
Split
Stream 2
Stream 1
Logical
operatorsContinuous Queries
Data Stream Management System (DSMS)
Infrastructure
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 6 / 31
Apache Storm
The structural view and logical view of a Storm Cluster
B1
B2
A1C2
C1
A1
B1
B2
C1
C2
A B C
Topology View
Task View
Worker View
Operator Parallelization
Task Scheduling
Cluster View
Nimbus(Master)
ZookeeperZookeeper
Zookeeper
Task
Worker Process
Executor
Supervisor
Worker Node
(a) (b)
Structural view Logical view
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 7 / 31
Task Scheduling in Apache Storm
Static schedulersDefault scheduler
Assign executors as evenly as possible between all the workersRound-robin procedure
Spout
Bolt1
Bolt2
Bolt3
Storm Topology Storm Cluster
Worker Process
Supervisor
Bolt3
Task
Worker Process
Supervisor
Bolt2
Task
Worker Process
Supervisor
Bolt1
Task
Worker Process
Supervisor
Spout Task
Default Scheduling
Peng’s Resource Aware Scheduler (RAS) [1]
Dynamic schedulerThroughput-oriented schedulers [2, 3]Latency-oriented schedulers [4, 5, 6]Communication-reduction schedulers [7, 8, 9]
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 8 / 31
Issues with Existing Schedulers
Static assignment
Not able to tackle runtime changesRequire users’ intervention to specify resource requirements
Resource agnostic
Mismatch of task resource demands and node resource availabilityLeading to over/under utilisation
Load balancing design
Endeavour to distribute the workload as evenly as possibleUnable to perform resource consolidation when the input is smallCollocating multiple applications results in resource contention
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 9 / 31
Outline
1 BackgroundStream ProcessingApache StormIssues with the Existing Scheduler
2 Solution OverviewOur ApproachFramework Overview
3 Dynamic Resource-Efficient SchedulingProblem FormulationScheduling Algorithm
4 Evaluation
5 Related Work
6 Conclusions and Future Work
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 10 / 31
Our Approach
Dynamic Resource-efficient Scheduling
A user-transparent mechanism in DSMS to schedule streamingapplications as compact as possible, matching the resource demands ofstreaming tasks to the capacity of distributed nodes, so that fewerresources are consumed to achieve the same performance target
Core contributions:
Profile streaming tasks at runtime to get accurate resource demands
Model and solve the scheduling problem as a bin-packing variant toenable resource consolidation
Reschedule the application automatically with a MAPE loop
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 11 / 31
D-Storm Framework
The extended D-Storm architecture
Nimbus
Worker Node Worker Node
Supervisor Supervisor
Topology Adapter
D-Storm Scheduler
Zookeeper
Task
Task Wrapper
Task
Task Wrapper
Task
Task Wrapper
Task
Task Wrapper
Worker Process Worker Process
System Analyser
Scheduling Solver
Control Flow
Data Stream Flow
Metric FlowCustom Scheduler
Task Wrapper:
A middle tier between task andexecutor abstractionsObtain task CPU usagesLog communication traffics
System Analyser:
A boundary checker on metricsDetermine whether the currentsystem state is normalTwo abnormal states:
Unsatisfactory performanceConsolidation required
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 12 / 31
D-Storm Framework
The extended D-Storm architecture
Nimbus
Worker Node Worker Node
Supervisor Supervisor
Topology Adapter
D-Storm Scheduler
Zookeeper
Task
Task Wrapper
Task
Task Wrapper
Task
Task Wrapper
Task
Task Wrapper
Worker Process Worker Process
System Analyser
Scheduling Solver
Control Flow
Data Stream Flow
Metric FlowCustom Scheduler
Task Wrapper:
A middle tier between task andexecutor abstractionsObtain task CPU usagesLog communication traffics
System Analyser:
A boundary checker on metricsDetermine whether the currentsystem state is normalTwo abnormal states:
Unsatisfactory performanceConsolidation required
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 12 / 31
D-Storm Framework
The extended D-Storm architecture
Nimbus
Worker Node Worker Node
Supervisor Supervisor
Topology Adapter
D-Storm Scheduler
Zookeeper
Task
Task Wrapper
Task
Task Wrapper
Task
Task Wrapper
Task
Task Wrapper
Worker Process Worker Process
System Analyser
Scheduling Solver
Control Flow
Data Stream Flow
Metric FlowCustom Scheduler
Task Wrapper:
A middle tier between task andexecutor abstractionsObtain task CPU usagesLog communication traffics
System Analyser:
A boundary checker on metricsDetermine whether the currentsystem state is normalTwo abnormal states:
Unsatisfactory performanceConsolidation required
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 12 / 31
D-Storm Framework
The extended D-Storm architecture
Nimbus
Worker Node Worker Node
Supervisor Supervisor
Topology Adapter
D-Storm Scheduler
Zookeeper
Task
Task Wrapper
Task
Task Wrapper
Task
Task Wrapper
Task
Task Wrapper
Worker Process Worker Process
System Analyser
Scheduling Solver
Control Flow
Data Stream Flow
Metric FlowCustom Scheduler
Scheduling Solver:
Invoked by the System Analyserreporting an abnormal systemstatePerform scheduling calculation,devising a new plan
D-Storm Scheduler:
Implement the ISchedulerinterfacePut the new plan into effect
Topology Adapter:
Mask changes made for profiling
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 13 / 31
D-Storm Framework
The extended D-Storm architecture
Nimbus
Worker Node Worker Node
Supervisor Supervisor
Topology Adapter
D-Storm Scheduler
Zookeeper
Task
Task Wrapper
Task
Task Wrapper
Task
Task Wrapper
Task
Task Wrapper
Worker Process Worker Process
System Analyser
Scheduling Solver
Control Flow
Data Stream Flow
Metric FlowCustom Scheduler
Scheduling Solver:
Invoked by the System Analyserreporting an abnormal systemstatePerform scheduling calculation,devising a new plan
D-Storm Scheduler:
Implement the ISchedulerinterfacePut the new plan into effect
Topology Adapter:
Mask changes made for profiling
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 13 / 31
D-Storm Framework
The extended D-Storm architecture
Nimbus
Worker Node Worker Node
Supervisor Supervisor
Topology Adapter
D-Storm Scheduler
Zookeeper
Task
Task Wrapper
Task
Task Wrapper
Task
Task Wrapper
Task
Task Wrapper
Worker Process Worker Process
System Analyser
Scheduling Solver
Control Flow
Data Stream Flow
Metric FlowCustom Scheduler
Scheduling Solver:
Invoked by the System Analyserreporting an abnormal systemstatePerform scheduling calculation,devising a new plan
D-Storm Scheduler:
Implement the ISchedulerinterfacePut the new plan into effect
Topology Adapter:
Mask changes made for profiling
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 13 / 31
Outline
1 BackgroundStream ProcessingApache StormIssues with the Existing Scheduler
2 Solution OverviewOur ApproachFramework Overview
3 Dynamic Resource-Efficient SchedulingProblem FormulationScheduling Algorithm
4 Evaluation
5 Related Work
6 Conclusions and Future Work
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 14 / 31
Problem Formulation
Table: Symbols used for dynamic resource-efficient scheduling
Symbol Description
n The number of tasks to be assignedτi Task i , i = 1, ..., nm The number of available worker nodes in the clusterνi Worker node i , i = 1, ...,m
W νic CPU capacity of νi , measured in a point-based system
W νim Memory capacity of νi , measured in Mega Bytes (MB)
ωτic Total CPU requirement of τi in pointsωτim Total memory requirement of τi in Mega Bytes (MB)ρτic Unit CPU requirement for τi to process a single tupleρτim Unit memory requirement for τi to process a single tupleξτi ,τj The size of data stream transmitting from τi to τjΘτi The set of upstream tasks for τiΦτi The set of downstream tasks for τiκ The volume of inter-node traffic within the cluster
Vused The set of used worker nodes in the clustermused The number of used worker nodes in the cluster
minimise κ(ξξξ,xxx) =∑
i,j∈{1,..,n}
ξτi ,τj (1 −∑
k∈{1,..,m}
xi,k ∗ xj,k)
subject tom∑
k=1
xi,k = 1, i = 1, ..., n,
n∑i=1
ωτic xi,k ≤ W νkc k = 1, ...,m,
n∑i=1
ωτim xi,k ≤ W νkm k = 1, ...,m,
(1)
where xxx is the control variable that stores the taskplacement in a binary form: xi ,k = 1 if and only iftask τi is assigned to machine νk .
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 15 / 31
Scheduling Algorithm
Algorithm 1: The multidimensional FFD heuristic scheduling algo-rithmInput: A task set ~τ = {τ1, τ2, . . . , τn} to be assignedOutput: A machine set ~ν = {ν1, ν2, . . . , νmused
} with each machinehosting a disjoint subset of ~τ , where mused is the number ofused machines
1 Sort available nodes in descending order by their resource availabilityas defined in Eq. (4)
2 mused ← 03 while there are tasks remaining in ~τ to be placed do4 Start a new machine νm from the sorted list;5 if there are no avaiable nodes then6 return Failure
7 Increase mused by 18 while there are tasks that fit into machine νm do
9 foreach τ ∈ ~τ do10 Calculate %(τi , νm) according to Eq. (5)
11 Sort all viable tasks based on their priority
12 Place the task with the highest %(τi , νm) into machine νm13 Remove the task from ~τ14 Update the remaining capacity of machine νm
15 return ~ν
Algorithm Design:
Based on greedy heuristicLarge machines are utilised firstTask priority updated dynamicallySeek to increase the sum ofinternal communications at thecurrent node
Complexity Analysis:
Line 1 requires at most quasilineartime O(mlog(m))Line 8 to Line 14 will be repeatedfor at most n timesLine 10 consumes linear time of nThe worst case complexity:O(mlog(m) + n2log(n))
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 16 / 31
Scheduling Algorithm
Algorithm 2: The multidimensional FFD heuristic scheduling algo-rithmInput: A task set ~τ = {τ1, τ2, . . . , τn} to be assignedOutput: A machine set ~ν = {ν1, ν2, . . . , νmused
} with each machinehosting a disjoint subset of ~τ , where mused is the number ofused machines
1 Sort available nodes in descending order by their resource availabilityas defined in Eq. (4)
2 mused ← 03 while there are tasks remaining in ~τ to be placed do4 Start a new machine νm from the sorted list;5 if there are no avaiable nodes then6 return Failure
7 Increase mused by 18 while there are tasks that fit into machine νm do
9 foreach τ ∈ ~τ do10 Calculate %(τi , νm) according to Eq. (5)
11 Sort all viable tasks based on their priority
12 Place the task with the highest %(τi , νm) into machine νm13 Remove the task from ~τ14 Update the remaining capacity of machine νm
15 return ~ν
Algorithm Design:
Based on greedy heuristicLarge machines are utilised firstTask priority updated dynamicallySeek to increase the sum ofinternal communications at thecurrent node
Complexity Analysis:
Line 1 requires at most quasilineartime O(mlog(m))Line 8 to Line 14 will be repeatedfor at most n timesLine 10 consumes linear time of nThe worst case complexity:O(mlog(m) + n2log(n))
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 16 / 31
Outline
1 BackgroundStream ProcessingApache StormIssues with the Existing Scheduler
2 Solution OverviewOur ApproachFramework Overview
3 Dynamic Resource-Efficient SchedulingProblem FormulationScheduling Algorithm
4 Evaluation
5 Related Work
6 Conclusions and Future Work
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 17 / 31
Experiment Setup
Nectar IaaS Cloud
16 worker nodes: 2 VCPUs, 6GB RAM and 30GB disk1 Nimbus, 1 Zookeeper, 1 Kestrel node
Extended on Apache Storm v1.0.2, with Oracle JDK 8, update 121
Two test applications
Synthetic linear applicationTwitter Sentiment Analysis topology
Profiling environment
Topology
D-Storm Cluster
Message Queue
Message Generator
Metric Reporter
Data Stream Flow
Metric Flow
VMs
Performance Monitor
Op Op Op Op
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 18 / 31
Experiment Setup
Synthetic linear application
3 synthetic bolts concatenated in serialProduce different patterns of resource consumption, such as CPUbound and I/O bound computations.
Twitter Sentiment Analysis topology
11 operators constituting a tree-like topologySentimental score calculated using AFFINN, a list of words associatedwith pre-defined sentiment values.
Table: Evaluated configurations and their values
Configuration Value
Cs (synthetic topology) 10, 20, 30, 40Ss (synthetic topology) 1, 2, 3, 4Ts (synthetic topology) 4, 8, 12, 16Number of tasks (realistic application) 11, 21, 31, 41
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 19 / 31
Applicability Evaluation
Research Question
Whether D-Storm is applicable to different types of streaming applicationsand capable of reducing the total amount of inter-node communication?
10 20 30 40-50
-40
-30
-20
-10
0
10
20
30
40
50
Rel
ativ
e ch
ange
s in
inte
r-no
de c
omm
unni
catio
n
D-StormStatic RAS
(a) Varying Cs
(Sythetic App)
1 2 3 4-50
-40
-30
-20
-10
0
10
20
30
40
50
Rel
ativ
e ch
ange
s in
inte
r-no
de c
omm
unni
catio
n
D-StormStatic RAS
(b) Varying Ss
(Sythetic App)
4 8 12 16-50
-40
-30
-20
-10
0
10
20
30
40
50
Rel
ativ
e ch
ange
s in
inte
r-no
de c
omm
unni
catio
n
D-StormStatic RAS
(c) Varying Ts
(Sythetic App)
11 21 31 41-50
-40
-30
-20
-10
0
10
20
30
40
50
Rel
ativ
e ch
ange
s in
inte
r-no
de c
omm
unni
catio
n
D-StormStatic RAS
(d) Varying numberof tasks (Real App)
Figure: The relative change of the inter-node communication, with the baselineproduced by the Default Storm scheduler
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 20 / 31
Cost-Efficiency Evaluation
Research Question
How is D-Storm performing when the input load decreases and the clusteris under-utilized?
4000 3000 2000 10000123456789
1011121314151617181920
Num
ber
of u
sed
wor
ker
node
s by
D-S
torm
CPU Intensive Synthetic AppI/O Intensive Synthetic App
(a) The number ofused nodes
4000 3000 2000 10000
0.10.20.30.40.50.60.70.80.9
11.11.21.31.41.51.61.71.81.9
2
Mon
etar
y co
st s
aved
(un
it: A
UD
/hou
r)
CPU Intensive Synthetic AppI/O Intensive Synthetic App
(b) The savedmonetary cost
Figure: Cost efficiency analysis of D-Storm scheduler as the input load decreases.Fig. 2b is calculated based on the price of m1.medium instances in the AWSSydney region
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 21 / 31
Scheduling Overhead Evaluation
Research Question
How long does it take for D-Storm to schedule relatively large streamingapplications?
Table: The time consumed in creating schedules by different strategies (unit:milliseconds)
SchedulersTest Cases Parallelism Intensive Synthetic App
Ts=4 Ts=8 Ts=12 Ts=16
D-Storm 16.4 16.2 16.8 17.2Static Scheduler 5.2 5.7 5.5 5.9Default Scheduler 0.72 0.77 0.75 0.79
Twitter Sentiment Analysis
Nt=11 Nt=21 Nt=31 Nt=41
D-Storm 13.4 13.6 13.8 13.9Static Scheduler 3.41 3.61 3.53 3.91Default Scheduler 0.61 0.62 0.65 0.62
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 22 / 31
Outline
1 BackgroundStream ProcessingApache StormIssues with the Existing Scheduler
2 Solution OverviewOur ApproachFramework Overview
3 Dynamic Resource-Efficient SchedulingProblem FormulationScheduling Algorithm
4 Evaluation
5 Related Work
6 Conclusions and Future Work
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 23 / 31
Related Work
In comparison with existing schedulers, we conclude that our work isdynamic resource-aware, communication-aware, self-adaptive,user-transparent and cost-efficient.
Table: Related work comparison
AspectsRelated Works Our
[7] [1] [8] [9] [4] [10] [2] Work
Dynamic Y N Y Y Y N Y YResource-aware N Y N N Y Y N YCommunication
Y N Y Y N Y Y Y-awareSelf-adaptive Y N Y Y N N Y YUser-transparent N N Y Y N N N YCost-efficient N Y N N Y N N Y
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 24 / 31
Outline
1 BackgroundStream ProcessingApache StormIssues with the Existing Scheduler
2 Solution OverviewOur ApproachFramework Overview
3 Dynamic Resource-Efficient SchedulingProblem FormulationScheduling Algorithm
4 Evaluation
5 Related Work
6 Conclusions and Future Work
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 25 / 31
Conclusions
We proposed a resource-efficient algorithm for scheduling streamingapplication with bin-packing formulation.
We implemented D-Storm, a prototype scheduler for algorithmvalidation.
The compact scheduling strategy leads to the reduction of resourceusages as well as the minimisation of inter-node communication
We adjust the scheduling plan to the runtime changes whileremaining sheer transparent to the upper-level application logic.
Outlook
Use meta-heuristics to find a better solutionTake the network characteristics into account during the schedulingprocess, placing intensive task communications on links with higherbandwidth.
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 26 / 31
References I
B. Peng, M. Hosseini, Z. Hong, R. Farivar, and R. Campbell,“R-storm: Resource-aware scheduling in storm,” in Proceedings of the16th Annual Conference on Middleware, ser. Middleware ’15. ACMPress, 2015, pp. 149–161.
C. Li, J. Zhang, and Y. Luo, “Real-time scheduling based onoptimized topology and communication traffic in distributed real-timecomputation platform of storm,” Journal of Network and ComputerApplications, vol. 87, no. 3, pp. 100–115, jun 2017.
T. Buddhika, R. Stern, K. Lindburg, K. Ericson, and S. Pallickara,“Online scheduling and interference alleviation for low-latency,high-throughput processing of data streams,” IEEE Transactions onParallel and Distributed Systems, vol. 28, no. 12, pp. 3553–3569, Dec2017.
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 28 / 31
References II
D. Sun, G. Zhang, S. Yang, W. Zheng, S. U. Khan, and K. Li,“Re-stream: Real-time and energy-efficient resource scheduling in bigdata stream computing environments,” Information Sciences, vol. 319,pp. 92–112, oct 2015.
D. Sun and R. Huang, “A Stable Online Scheduling Strategy forReal-Time Stream Computing over Fluctuating Big Data Streams,”IEEE Access, vol. 4, pp. 8593–8607, 2016.
A. Chatzistergiou and S. D. Viglas, “Fast heuristics for near-optimaltask allocation in data stream processing over clusters,” inProceedings of the 23rd ACM International Conference on Informationand Knowledge Management, ser. CIKM ’14. ACM Press, 2014, pp.1579–1588.
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 29 / 31
References III
L. Aniello, R. Baldoni, and L. Querzoni, “Adaptive online schedulingin storm,” in Proceedings of the 7th ACM International Conference onDistributed Event-Based Systems, ser. DEBS ’13. ACM Press, 2013,pp. 207–218.
L. Fischer and A. Bernstein, “Workload scheduling in distributedstream processors using graph partitioning,” in Proceedings of the2015 IEEE International Conference on Big Data. IEEE, 2015, pp.124–133.
J. Xu, Z. Chen, J. Tang, and S. Su, “T-storm: Traffic-aware onlinescheduling in storm,” in Proceedings of the 2014 IEEE 34thInternational Conference on Distributed Computing Systems, ser.ICDCS ’14. IEEE, 2014, pp. 535–544.
Xunyun Liu et al. (CLOUDS Lab) ICPADS 2017 December, 2017 30 / 31