Accounting and Allocation Management with Gold

Scott Jackson

Moab·Con 2010

Allocation ManagerGold is an open source allocation system that tracks and charges for resource usage on High Performance Computers.

Jan-99 Feb-99 Mar-99 Apr-99 May-99 Jun-99 Jul-99

Alloc6

Alloc5

Alloc4

Alloc3

Alloc2

Alloc1

.

10,000 NH

10,000 NH

10,000 NH

10,000 NH

10,000 NH

10,000 NH

Ensures resources are used according to mission plan

Tracks resource utilization Allows for insightful capacity

planning Facilitates resource sharing

between organizations Used in production at dozens of

sites

Gold in a nutshell: A resource billing system

Gold tracks and manages resource usage on High Performance Computers. It behaves like a bank in which deposits are made into accounts and charges occur at the moment of job completion. Reservations prevent oversubscription.

An allocation managerCredits can be allocated to specific projects, users and machines.

Timeframes can be given for the expenditure of the credits enabling allocation cycles, capacity planning and use-it-or-lose it.

An accounting systemGold maintains past state and a complete transaction history. It can track

and maintain arbitrary custom accounting statistics. An information service

Custom objects and attributes can be dynamically defined and manipulated.

Setting up Gold

Define Users, Projects, MachinesAdd Users to ProjectsSetup Default Projects (optional)Make Deposits into Accounts

AccountsProjects

UsersMachines

Allocations

Universeof

Projects

Account: 1234Name: Chemistry

Universeof

Users

Universeof

Machines

Account: 1234Name: Chemistry

TimePeriod Allocation CreditLimit

F1Q10 250,000 0F2Q10 250,000 0F3Q10 250,000 0F4Q10 250,000 0FY10 0 20,000

Allocations An allocation is an allotment of computational resource credits

for use by specific projects or users during a specified time period.

When jobs complete, projects are charged and resource usage recorded. Jobs are prevented from starting unless they have an active allocation with a sufficient balance.

Limitations: Does manage short-term usage, does not impact scheduling priority.

Project Users Machine Time Period Balance

Biology Amy, Bob Colony FY2005 100,000

chemistry Bob, Dave Colony FY2005 150,000

Reservations and Dynamic Charging

Accounting Manager

(Gold)

0

2

1

4

3

5

6

Deposit

0 Deposits are made in Account1 A Job is Submitted2 A Quote is Requested

Quote

3 A Reservation is Made

Scheduler(Moab)

4 The Job is allowed to Start

Reservation

5 The Job Completes

Resource Manager(Torque, LL, LSF)

6 The Reservation is Removed

Reservation

and a Charge is Issued

Charge

Capacity & Workload Planning:The Brick Approach Controlling project usage is key to capacity and workload

planning. The use of regularly expiring allocations allows you to

establish a project cycle and use the brick approach to allocation management.

Now- 1 Qtr + 3 Qtr+ 2 Qtr+ 1 Qtr- 3 Qtr - 2 Qtr + 4 Qtr

100 % Capacity

Year-end Resource Exhaustion It is common to see projects get off to a slow start and then

have a surge of activity at the end when results are due. Without careful management of expectations, this can be a

cause of considerable anxiety.

100% Machine Capacity

Project A’s Allotment

Project A’s Demand

Project Period

Expiring Allocations

Jan-99 Feb-99 Mar-99 Apr-99 May-99 Jun-99 Jul-99

Alloc6

Alloc5

Alloc4

Alloc3

Alloc2

Alloc1

.

10,000 NH

10,000 NH

10,000 NH

10,000 NH

10,000 NH

10,000 NH

Jan-99 Feb-99 Mar-99 Apr-99 May-99 Jun-99 Jul-99

Alloc1

.

60,000 NH

Jan-99 Feb-99 Mar-99 Apr-99 May-99 Jun-99 Jul-99

Alloc7

Alloc6

Alloc5

Alloc4

Alloc3

Alloc2

Alloc1

.

5,000 NH

10,000 NH

10,000 NH

10,000 NH

10,000 NH

10,000 NH

5,000 NH

Jan-99 Feb-99 Mar-99 Apr-99 May-99 Jun-99 Jul-99

Alloc6

Alloc5

Alloc4

Alloc3

Alloc2

Alloc1

.

10,000 NH

10,000 NH

10,000 NH

10,000 NH

10,000 NH

10,000 NH

ChargeRates Consumable Resource Charge Rates — Consumable Resource Charge Rates define how much it costs

per unit of time to use a consumable resource like processors, memory, etc. Usage Charge Rates — Usage Charge Rates define how much to charge for metrics of total resource

usage such as CpuTime, Licenses used, Power consumed (in Watts), etc. Name-Based Multiplier Rates — Name-Based Multiplier Rates are quality based charge rates which

apply a multiplicative charge factor based on a quality of the job such as QOS, nodetype, queue, user, time of day, etc.

Value-Based Multiplier Rates — Value-Based Multiplier are scaled multipliers which apply a multiplicative charge factor based on a numeric scaling factor.

Type Name Rate Description

Resource Processors 1 Processor Charge

Resource Memory .001 Memory in MB

Usage Power .05 Watts

Usage CpuTime 1 Actual CPU

QOS Premium 2 Hi Prio, Preemptor

QOS Normal 1 Regular Prio

QOS Bottomfeeder .5 Lo Prio, Preemptee

Other Gold Features

Flexible Charging

Credit and Debit ModelsGuaranteed Quotes

Powerful Querying

Other Gold Features

Strong Security

Historical JournalingTransparency Features

Dynamically Customizable

Web-based GUI

Nested AccountsUniversity

Biology

Actinides

Chem 201

Physics

Chem 101

Chemistry

Workshop

252525

4040

5040

End of Life Spending Spree Offline banks cannot strictly enforce allocations Without reservations, more jobs can get started

than allocation balance can support.

Job

DebitAccounts

Job

DebitAccounts

Sneak Peek -- ChargeRatesType Name Instance Rate Description---------- ---------- --------- ---- ------------------------------Resource Processors 10 Value Based Resource (VBR)Service License Abacus 10 Name Based Resource (NBR)Usage Power 500 Value Based Usage (VBU)Feature Feature Telescope 5 Name Based Usage (NBU)Quality QOS Premium 2 Name Based Multiplier (NBM)Multiplier Discount 0.5 Value Based Multiplier (VBM)Fee Shipping 100 external Value Based Usage (xVBU)Token Zone Asia 400 external Name Based Usage (xNBU)*(Disk) User scottmo 10 Multi-dimensional VBR (MVBR)

The multi-dimensional Resource is a resource value that changes on a per-attribute basis, and is assumed if the Type is anything other than the known types. For example, the above line illustrates a disk resource fee of 10 for User=scottmo, whereas a different value might be applied for User=fred.

Total Charge = ((((sum of all NBR and (M)VBR*value) times walltime) + (sum of all NBU and VBU*value)) times (product of all NBM and VBM*value)) + (sum of all xNBU and xVBU*value)