Optimized your sql server operation using big data echo system

All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)

1

Optimized your sql server operation using big data echo system


2

Agenda

• Introduction

• Big data– The bad the good & the ugly

• Open source projects

• Puppet • Grafana• Logstash, ElasticSearch and Kibana

• Summary


3

Agoda.com

• Agoda.com is one of the world`s leading online hotel booking platforms.

• We work with hundreds of thousands of hotels worldwide to offer discounted hotel rates to customers.


4

Agoda Production environment

• 600 instances

• Support 7 data engines • Sql server • Cassandra• Verica• Cubes• Elastic search• Spark • Hadoop

• 4 data centers


5

Agoda data team


6

Data Operation Roles & Responsibilities

• Installations & upgrade • Maintenance

• Monitoring - active , proactive

• Performance Tuning

• Deploy database changes


7

Stairway (pyramid) to heaven

Hardware

Operating system

Data engine

Application


8

The bad the good & the ugly


9

Puppet

• Puppet is an open source data center automation and configuration management framework.

• Scalable to thousands of nodes

• Web-based dashboard for reporting

• Hundreds of community developed modules

• puppetlabs/sqlserver


10


11

Puppet – use cases

• Install

• Upgrade

• Migration –Storage , server

• Deployment lifecycle – safe & fast


12

No upload, No cry

Establish a baseline

Identify bottlenecks

OptimizeMeasure performance

Repeat (if desired)


13

Grafana

• Grafana provides a powerful and elegant way to create, explore, and share dashboards,

• Open source

• Real time

• Feature Highlights • Annotations• Template Variables• Rich Graphing

• play.grafana.org


14

Grafana architecture


15

Collect Source

• Performance monitor• Query • XE • Error logs • JVM

MethodUnix – collectDWindows - Graphite-PowerShell

Architecture Policy based Central collector vs local collector


16

Cluster management


17

Scale using replication

Master server

DC 1 DC 2 DC 3 DC 4

20xfront end

2xRPL 2xRPL 2xRPL

2xRPL

20x front end

12x front end

12x front end


18

Monitor Replication


19

Replication – performance monitor counters Agent Performance object Counter Description

Log Reader Agent SQL Server: Replication Logreader

Logreader: Delivered Cmds/sec

The number of commands per second delivered to the Distributor.


Logreader: Delivered Trans/sec

The number of transactions per second delivered to the Distributor.


Logreader: Delivery Latency The current amount of time, in milliseconds, elapsed from when transactions are applied at the Publisher to when they are delivered to the Distributor.

Distribution Agent SQL Server: Replication Dist. Dist: Delivered Cmds/sec The number of commands per second delivered to the Subscriber.

Distribution Agent SQL Server: Replication Dist. Dist: Delivered Trans/sec The number of transactions per second delivered to the Subscriber.

Distribution Agent SQL Server: Replication Dist. Dist: Delivery Latency The current amount of time, in milliseconds, elapsed from when transactions are delivered to the Distributor to when they are applied at the Subscriber.


20

Logstash, ElasticSearch & KibanaLogstash

• Centralize data processing of all types• Normalize varying schema and formats• Quickly extend to custom log formats

• ElasticSearch

• Search server based on Lucene.• Scaleable

• Kibana

• Flexible analytics and visualization platform• Real-time summary and charting of streaming data


21

Logstash, ElasticSearch & Kibana


22

Elastic Search – Use cases

• Application logs

• Data engine logs

• OS logs

• Hardware logs - network , storage

• Free text search


23

Kapunkap


24

We are looking for new talents!

• Are you ready to work in a fast paced environment, with opportunity for

career progression?

• Do you want to be part of a multicultural team, with colleagues from all

around the world?

• Do you want to utilize and improve your foreign language skills further?

• Are you ready to take up challenges and give prompt solutions for

customers with their requests?

Technology

Optimized your sql server operation using big data echo system