Proof of Concepts and Benchmarks etc.

Definitions

• Benchmark– The customer may know the product works, but are we the

best??– Maybe speed rather than facilities

• Proof-of-Concept (PoC)– Does the product work ? (basic tick in the box)– Can it do what I want it to do ? (facilities)– Can it handle my data ? (Volumes)

Differences

• Generally there is (usually) a greater sense of urgency with a Benchmark

• A benchmark is practically always competitive

• A rule of thumb could beProof of Concept - Customer is with you

Benchmark - Customer is not generally with you

Step 1

Benchmark as a Last Resort

Benchmarks can be very risky

• Competition uses losses as proof points in future deals, advertisements etc.– Every loss can require Five wins to compensate

• Prematurely exposes Minor Shortcomings– Each may be minor and easily dealt with, but together, may

contribute to customer feeling “buyer’s remorse” - before they buy!

Questions

• Do we do the work ?– Can we do this technically ?– Do we want to do this commercially ?

• The first question is ours to answer

• The second we can give “advise” on - but should not answerHe who fights and runs away, lives to fight another day

Resource-Intensive

• Other Vendors May Be Able to Throw More Bodies at it– This definitely was Oracle’s strategy with Early Sybase

System 10

• Ensure Adequate Resource Commitment From Sybase and Customer– Play up the partnership commitment and long-term value to

the customer– Inadequate resources and preparation almost guarantees

failure

Requirements

• Plan, Plan and Plan

• Resources– Technical buy in from both the Customer and your company– Your time– The plan - including decision points

• Hardware/Software Availability

The Plan

• What we must have to make this work– People (customer and company), computers, software

• How long is this going to take (multiply by 2 at least!)

• Decision Points– Where can we stop, survey and decide to continue or

cut our losses

Time

Richard’s First Law of BenchmarksEverything has to be done 4 times

• 1st Time - It will not work

• 2nd Time - It partially works - then crashes

• 3rd Time - It works, but no-one believes it, and you forgot to time it

• 4th Time - It works and you did time it.

Resources - Your company

• Your Company– Your time– Technical Support– Sales Person– Management buy-in– Technical stand-in

The Sales Person (yes they do have uses)

• This person is worth their weight in gold– Well… Maybe silver

• Their job is to shield you from interference from– The Customer– The Company

• Their job is also to get you more resources or time if you need it

Resources - The Customer

• Customer– Technical Assistance

Someone who KNOWS the system (not the guy the Technical Director first thought of)

– Data and Schema (or at least some form of data definition)– Queries, or at least list of questions– Timescale (when is the finish date for the project)

Customer Technical Assistance

• You must have someone from the Customer at your side during the PoC

• Phone calls to the Customer eat a lot of time• Trying the find the “right” person to speak to takes

even longer!Beware the phrase “Oh, didn’t I mention that”

• Treat all given information as unproven (if not actually wrong)

Success/Failure

SUCCESS CRITERIASUCCESS CRITERIA

• Without the above– How do we know if we have failed?– How do we know if we have succeeded?– What is the next step if we have succeeded?

• Criteria also mean we have a target to aim for, and we limit the work required

Time and Success

Good enough, in time = good

Perfect, too late = bad

• You must stick to the timing plan and aim ONLY for the success criteria

• “Wouldn’t it be nice if we could run …” is the most horrible phrase ever to be heard in a benchmark

Good Benchmark Practice - 1

• Take Notes– Have a complete list of what you did and when you did it.– It will save time in the long run and will allow you to write up

the project

• Script Everything you do on the system– You WILL have to do everything more than once!

Good Benchmark Practice - 2

• Be aware of the clock– If the timing is looking tight, or impossible– Discuss with the Sales Person, he may be able to buy you

more time, or extra resources

• Don’t be afraid to ask for help– You cannot do everything yourself– You do not know everything– Ask - not asking means lost time and lost sales

Finally

• OK we have– A Plan– Resources– Customer and Management Buy in– A Sales Person– A target– Computers and Time to do it

• Let’s go do Step 2

Step 2

The System

• Processors

• Memory

• Disk (sub-system) inc. RAID

• Operating System

• IQ 12

• Other Software

The Free Hand

• If we are not constrained and have a free hand– If we oversize - the customer may consider IQ is too

hardware hungry– If we undersize then the queries will run slowly or not at all

• We have to get this about right

CPU’s

• Proof of Concept– More is better

• Benchmark– IQ 12 is not parallel so if competitive with small number of

users - small number of CPUs– If competitive, with large number of users - as many CPUs

as you can get in the box!

Memory

• More is better

• We can always use more memory

• Consider 15MB per user (that is a very bad generalisation - but almost accurate!)

Disk - 1

• Two sorts– Simple Disk– Disk Farm (Storage Array)

• RAID x

• Suggestion– Disk Farm - as many spindles as possible– RAID 0/1 (Mirror/Stripe) - fast and reliable

Disk - 2

• How Much….• IQ Main

– 90% Raw - max.

• IQ Temp– 25% Raw - max.

• Staging Area– How long is piece of string…..– You made need more than one “copy” of the data

Disk Farm

• EMC, SSA, MTI etc.

• These may have complex set-up routines

• If possible, let the H/W supplier (or the customer) set up the disks

• Watch ! And Take Notes!

Other Hardware

• Extra Ethernet Adapters– 1 100Mbps is worth more than 10 10Mbps– 2 100Mbps is better still

• Dedicated LAN/Hardware– if not, be aware of other users - especially when running

timed tests

• Tape Units– Are you testing Backup?

Operating System

• The correct Revision to run IQ

• All the require OS Patches

• Has it been installed correctly– Do you need a Hardware Rep. To help with the install?– If this is a “system” benchmark, maybe you should plan a

hardware rep. To be on site

Software

• IQ– Have you the latest revision?– Have you read the release notes?– Are there any new EBFs?– Speak with Tech. Support PSE or Engineering get the latest

revision (that works!)

• Other Software– Replication Server, Distribution Director etc.– Are these all the latest revision?– Does all the software work together?

Step 3

Installation

• Install IQ

• Decide on IQ Page Size

• Build the Database

• Create the IQ Main Store

• Create the IQ Temp. Store

IQ Page Size

• 64 Kbytes, unless– Big database then 128 Kbytes– Big, Big database then 256 Kbytes– Not 512 – remember the bug…

Catalogue Store

• Nearly always forgotten

• More space needed for general “ASA” staging space

• If larger use RAW otherwise use Filesystem

• This store was intended to fit into memory

IQ Store Questions

• RAW or Filesystem– Unless there overwhelming reasons, and I can’t think of any,

then RAW

• Few Bigger, or Many Smaller– Many Smaller is better, but you may not have the choice

After Install and DB Create

• Test using sp_iqstatus– Is the database the correct size, did all the dbspaces create

OK

• Test using sp_iqcheckdb– If we have the time, let’s make sure that we have no errors

at this stage

• Re-check - are you sure we have enough space in the database?

Step 4

What to Do

• Create the tables

• Decide on the the “fast” indexes

• Create the “fast” indexes

• Decide on the HNG indexes

• Create the HNG indexes

• Test the installation

Table Creation

• Strip out ALL constraints except– PRIMARY KEY on single columns

– FOREIGN KEY on single columns

– UNIQUE

– IQ UNIQUE must must use

• Generally in a PoC or Benchmark do not use constraints or permissions

• In addition run everything from user DBA unless the customer has any real problems with this

Fast Index Decision

• A fast index is the primary performance index on an IQ system– Low Fast - Low Cardinality– High Group - High Cardinality– Join Columns – HG Index

• Cardinality breakpoint defined at around 1000-2000 for this case

For EVERY Column

• Is the column EVER going to be used for more than just projection?

• If the column fails the above test then do not waste time and space applying any more indexes on this column

Remember the column you do not index is the one that will be used as a search column come the day of the presentation!

Cardinality

• If the column is to have an index on it, decide on the cardinality

• You may not have this information• I use the WAG method, Wild Asses Guess• If you have the time and disk spaces create a High

Group on the column, load the data and perform a Select Distinct, this gets the exact cardinality (don’t do this on the 2.9 billion row table)

Fast Index

• For High Cardinality put a High Group

• For Low Cardinality put a Low Fast

• Warning : Treat the Customer information as unproven

“I never knew we had that many suppliers”

• If you have to drop and recreate the index so what? You did allow time in the plan for reloading the data - didn’t you?

High Non Group Index

• For EVERY Column that has a “fast” index

• Is this column going to be used in the following– range searches– between search– avg(), sum()– root string searches (like “Syb%”)

• If the answer is yes (regardless of the cardinality) add a High Non Group index

• Remember the IQ UNIQUE clause

Test

• Check by looking at the sys.sysobjects table in the catalogue server

• You did script everything didn’t you?

• We may have to retrofit indexes at a later time, but let’s TRY and get most of them built now

Step 5

Loading the data

• Configure the server for load

• Pre-fix the data

• Stage the data

• Load the data

• Test the installation

• Do it all again (Probably!)

Configure for Load

• Not much required here for IQ 12

• Ensure that you use sp_iqstatus to check that you have allocated the memory you thought you had

• Consider increasing Temp and decreasing Main

Where does the data come from

• Another database– Consider conversions here, unload, modify and reload may

be faster than CONVERT()– Generally UNIX commands like AWK and SED can run

quicker than CONVERT() and certainly are quicker than aggregate statements from general RDBMS products

– If you do an unload and reload, you will need staging space (twice the size of the data)

Flat Files

• This is where the most fun is• I will state as a “fact”

– Most of the time the customer cannot tell you what the input file format is exactly Print the file - in ASCII and HEX Row and column delimiters generally are killers

– The 7 millionth 400 thousandth record will be different from all the others - always

• Same advice on CONVERT() applies here• Remember load performance switches (row delimited

by etc.)

All Loads

• Script everything

• You will never, never succeed the first time

• Only test load with 10 to 100 rows of data, the load will fail that bit quicker than with 1 billion rows

• Load the biggest file first, it will take the longest time

During Load

• With IQ 12 we can load table simultaneously

• Remember not the saturate the server, I would suggest not loading more than 1 table at a time

• Time all the loads - especially the test loads– if 1000 rows took 1000 minutes then the 10 billion row load

will take a long time

Test the Load

• Count(*) on all tables, does this agree with the input files?

• Check the insert logs

• Check quality– Set temporary option public.‘row_count’ = 20– select * from table (all the tables)

• check numeric, do sum() and or avg()

Backup ?

• Maybe now we should run a backup

• Check the data size - a 2 T byte backup is not fast

• Is backup and restore part of the success criteria?

• If not do not spend the time to run it

Success Criteria

• Was the load or load time part of the success criteria

• Do we pass

• Do we have to do it again

• Maybe we need dedicated conversion and load programs

• If we have to pass this we have to spend the time….

Time and Panic

• This task is a (relatively) complex and intellectual task

• Benchmarking is (for a number of reasons) a high stress activity

• It will take long hours and you may be alone for long periods

• Isolation and stress are part of the job

Mistakes etc.

• Around 3:00 am when the load is not working real stress happens - I’ve been there, it is not nice

• Ask for help

• Re-read your notes (this is one reason you must write everything down)

• Failing everything else go back to the hotel for a sleep and a shower

Ask For Help

• I’ll say it again - louder

IF YOU HAVE PROBLEMS ASK FOR

HELP

Step 6

Running the Queries

• Set up the Server to Run Mode

• Test Run every Query

• Timed Runs

• Check the times

• Rerun

• Repeat above until either it works or you run out of time

Server Set-up

• Remove all the load “bits” - unless there are update queries in the test

• Get every byte of memory you can out of the server

• Do not generate a “Benchmark Special”

Test The Queries

• If they run first time, then fine

• If they don’t– Is there a SQL error?– Does the SQL not conform to what IQ thinks SQL is?– Are we missing a column of table?– Is there a bug in IQ?– Check release notes, bug lists, tech. support

First Runs

• Start the Server (ideally boot the machine)

• Run the query Timed

• Run the Query againTimed

• Run the Query again until the timings stabilise

• Redo for all the queries

Timings

• I consider the “stabilised” time for the query to the the “real” time for the query to execute.

• But keep this time, the slowest time and the fastest time - the customer may want a “range” rather than a single stake in the ground

During Runs

• Check memory with IQ Monitor– But only when you are not timing - the overhead is small, but

so are some winning margins

• The steady state timings can take a long time to achieve, keep your eye on the clock

Multi-User Tests

• Same criteria as single user tests, but record all the times

• These take longer - plan for re-running the tests

More Rules/Guidelines

• Investigate anomalous behaviour - but only if you have time - remember this is a test first and learning experience second

• Write everything down, all the tests must be repeatable, otherwise it looks like fraud

• Good Luck!

Final Step

The last slide

• Write down everything during the test

• Write up the project as a white paper

• Share the knowledge gained with all your co-workers

• What worked - and what didn’t

• What did you learn during the tests (except maybe never to do another benchmark again!)

The real last slide

Next job :-

• Prepare for the next Proof of Concept and Benchmark

• Maybe this time with a little more knowledge (thanks to the last PoC/BM)

Proof of Concepts - End