High Volume Scheduling and Job Management With PostgreSQL

8/10/2019 High Volume Scheduling and Job Management With PostgreSQL

1/52

High-Volume

Scheduling and Job

Management with

PostgreSQL

Leonardo Meira, Software Engineer

April 3rd, 2014


2/52

!Has been in production for years, runnin

hundreds and thousands of scripts simuand hundreds of millions of scripts in tot

!PostgreSQL is the basis for our high-vol

traditional queuing system!Log manager

!Open Source

System


3/52

! Context (What does Fiksu do?)

Agenda


4/52


!

The Problem

Agenda


5/52


!

The Problem

! The Solution

Agenda


6/52


!

The Problem

! The Solution

! Wrap-up

Agenda


7/52


!

The Problem

! The Solution

! Wrap-up

! Fiksus Open Source Projects

Agenda


8/52


!

The Problem

! The Solution

! Wrap-up


! Questions

Agenda


9/52

Fiksu is a mobile atechnology compa

Fiksu makes it ea

mobile app markeacquire the users

need to grow their

business.


10/52

!

!

The Problem

!

!

!

!

Agenda


11/52

!Massive data retrieval needs

!

Jobs running periodically, guaranteed to

cloud environment

!

Machine failures handled gracefully

!Load balancing

!

Quickly diagnose troubles when failures

Problem


12/52

!

!

! The Solution

!

!

!

Agenda


13/52

Network Application Framework

!Framework for building applicatio

!Schedules, runs and monitors pro

across a distributed network.

!

Like a distributed CRON but cloprepared

!A front-end UI to control it all


14/52

A cloud prepared distributed cron

!Efficient queue management of applica

!

Machine node redundancy

!

Application run dependencies

!Run restrictions preventing clashing sc

from executing at the same time

!machine or cluster wide clash check

!

Load Balancing


15/52

A nice UI

!Historical process runs

!Scheduling reports

!Debugging

!Log/Machine/Queue managemen

!Machine health


16/52

Based on Af

!Ruby on Rails scripting framework

!

Open sourced in 2013 (https://github.cpublic/af)

!Provides command line parsing

!

Tight integration with PostgreSQL

!Modified log4r to make log manageme

!

Strong application component module


17/52

Tight integration with Postgresql

!Automatic management ofpg_stat_activity.application_name

!Provided by our gem pg_application_name

!Helpers for advistory locking

!

Provided by our gem pg_advisory_locker!Bulk data management

!More of a supplement to activerecords lac

!Provided by our gem bulk data methods


18/52

Network Application Framework

! Open sourced in 2013 (https://github.com/fiksu-p

!

Has been in production for years, hundreds of macross hundreds of machines

! Manages script run times and distribution (think m

multi-machine/distributed cron)

!

Cloud Machine watchdog, alarming and logging

!AWS EC2 + RDS for ease

! RDS is of the PostgreSQL variety


19/52

Overview A running system

NAFDB

Runner

Script5

Script6

Script7

Script8

Runner

Script1

Script2

Script3

Script4

Script configuration

Script Schedules

Script Queues

Logs

Alarming

Yo

N

Runner Machine 1 Runner Machine 2 Servic


20/52

Runners

! One per machine

! Responsible for

! Schedule management

! Queuing of periodic jobs

! Job starting and process management

!

Queue/clash management

! Load balancing

! Machine watchdogging

!

Seamless code version upgrades


21/52

Runners start up

!Clean old processes

!

Remove invalid jobs

!Wind down other runner


22/52

Runners main loop

! If we have been marked down, kill all local jobs and exit

! If any local jobs have died, clean them up

! If we can start any new jobs, start them

! Mark ourselves last_seen_at so other runners know we are alive

! If no machine has checked schedules in the last minute

!

Mark ourselves as the leader

! Queue any scripts that need have asked to run in this time perio

! Mark any other runners down if they havent updated their last_s

minutes

! Mark that we checked the schedules


24/52

Machines UI


25/52

Table "naf.applications"

Column | ---------------------+------------

id | integer

created_at | timestamp w

updated_at | timestamp w

deleted | boolean

application_type_id | integer

command | text

title | text

short_name | text

log_level | text

Schema applications/schedulesTable "naf.application_schedules"

Column | Type |

--------------------------------------+-----------------------------+

id | integer |

created_at | timestamp without time zone |

updated_at | timestamp without time zone |enabled | boolean |

visible | boolean |

application_id | integer |

application_run_group_restriction_id | integer |

application_run_group_name | text |

application_run_group_limit | integer |

run_interval | integer |

priority | integer |

enqueue_backlogs | boolean

run_interval_style_id | integer

application_run_group_quantum | integer

A li ti UI


26/52

Applications UI

Application Sched les UI


27/52

Application Schedules UI


28/52

Runners Queue management

! Historical jobs in partitioned tables

!

old tables dropped every month or so

! queued jobs in their own table

! well describe why a normal queue is not sat

!

running jobs in their own table! in memory hot can be replaced by a Dyn

thing but postgresql does a great job for hun

machines


32/52


33/52

Affinities

!Think of them as puzzle pieces

!

Machines have affinity slots

!Slots can be required

!Applications have affinity tabs

!

An application can run on any machine that hthat match all of its tabs

!A machine will only run applications that hav

all of its required slots

Slot


34/52

Affinity parameters

!Machines have an allocate-able numb

slots of a specific affinity

!Applications will allocate a certain num

slots at run start (or not run on that ma

!Load balancing

Affi i i Di ( l i )


35/52

Affinities Diagram (puzzle pieces)

1

Machine1 App1 App1 App2 App3

1 1

Schema affinities


37/52

! When this machine has an empty run slot, fill itthe next job with the most priority from the que

! Has all of the required affinities of this machi

! This machine has all the affinities demanded

! Whose affinity parameters match allocable u

machine

! Is not restricted by the run group restrictions

! Whose prerequisites have completed

The Queue Fetcher

Agenda


38/52

!

!

!

! Wrap-up

!

!

Agenda



39/52


NAFDB

Runner Script5

Script6

Script7

Script8

Runner

Script1

Script2

Script3

Script4

Script configuration

Script Schedules

Script Queues

Logs

Alarming

Yo

N

Runner Machine 1 Runner Machine 2 Servic

Overview Runners


40/52

Overview Runners

Application 1

Application 2

Application 3

Application N

Machine 1

Runner 1

Machine2

Runner 2

Machine Y

Runner Y

Conclusion


41/52

Conclusion

!N/Af has been in production for years,

hundreds and thousands of scriptssimultaneously and hundreds of million

scripts in total

!PostgreSQL is the basis for our high-v

non-traditional queuing system

Agenda


42/52

!

!

!

!


!

Agenda

Gems


43/52

Gems

!N/Af

!

Af

!

Partitioned

!Bulk Data Methods

!PG Advisory Locker

!PG Application Name

N/Af


44/52

N/Af

!https://github.com/fiksu-public/naf

!

A network application framework that lPostgreSQL to deliver high volume, dis

and redundant job scheduling and man

Af


45/52

Af

! https://github.com/fiksu-public/af

!Application framework that supports:

!

Command line options integrated into instance (andvariables

! Logging via log4r

! PostgreSQL advisory locking via pg_advisor_locke

! PostgreSQL database connection updates via

pg_application_name gem! Threads and message passing

!Application components adding loggers and comm

options

Partitioned


46/52

Partitioned

!https://github.com/fiksu/partitioned

!

Adds assistance to ActiveRecord for manipu(reading, creating, updating) an ActiveRecord

that represents data that may be in one of m

database tables (determined by the Models d

!

Supports the creation and deletion of child tapartitioning support infrastructure.

!Supports bulk inserts and updates

Bulk Data Methods


47/52

Bulk Data Methods

!https://github.com/fiksu/bulk_data_me

!

MixIn used to extend ActiveRecord claimplementing bulk insert and update o

through {#create_many} and {#update

PG Advisory Locker


48/52

PG Advisory Locker

!https://github.com/fiksu/pg_advisory_lo

!

Helper for calling PostgreSQL functionpg_advisory_lock, pg_advisory_try_loc

pg_advisory_unlock


49/52


50/52

Agenda


51/52

!

!

!

!

!

! Questions

g

Thank You!


52/52

Want to talk?

[email protected]

www.fiksu.com

@fiksu

Learn more:

https://github.com/fiksu-public/naf

Documents

High Volume Scheduling and Job Management With PostgreSQL