Keynote Benchmark

Making a Site,and Checking it Twice

A Guide to Preparing a Web site for the Holiday Rush

f any retailer needs proof of the importance of load testing, they need only look to the headlines around the launch of the new Apple

iPhone 4. All day on June 15, Apple and AT&T’s online stores went down and out. Tech bloggers spent hours trying to place online

orders, and hours more on hold with customer service. An official announcement from AT&T called it “the busiest online sales day in

AT&T history.” Yet despite the problems, they managed to sell out their first day’s allotment before the day was out.I

2

Retailers dream of having such problems as rolling out a new blockbuster and taking orders for 600,000 units in one day. But you can count on one hand the number of sellers who command such fanatic loyalty and patience. For the rest, site outages, freezes and checkout failure mean lost revenue that will likely never be made up — and those losses mount minute by minute, hour by hour, especially during the peak holiday season, when most retailers rack up most of their annual business. During those critical four to six weeks at the end of the year, when the site breaks, the bottom line breaks, too. So: it’s July or later as you’re reading this. The biggest, most successful online retailers have been prepping their sites for the holidays for six or seven months now. The next tier started in June. Retailers who have not started to stress test their sites could be looking at many sleepless nights come the last months of the year, and potentially serious hits to their 2010 revenues.

ONLINE MAKES THE BOTTOM LINEThere’s no underestimating the importance of

e-commerce to overall retail success,

particularly during the crucial holiday season.

Every year from 2000 through 2007, online

sales posted double-digit growth, handily

outpacing in-store sales.i And last year, as

retail struggled to recover from its worst

holiday ever (in 2008), online holiday sales

grew at four times the rate of total retail

growth — though admittedly modest growth,

at 4 percent for online sales,ii and just 1

percent for total retail.iii (For the year,

e-commerce sales grew 2 percent, while overall

retail sales fell by nearly 3 percent, according

to the U.S. Department of Commerce.)

With online playing such a critical

role in holiday sales, it is impossible to ignore

the real bottom-line costs in lost revenue of

site slowdowns and outages. Site metrics and

sales data show the revenue value of every site

visitor, and site testing can show the actual

cash lost when visitors abandon the site

because of poor performance. As the

accompanying chart shows, using actual

results, a traffic spike 25 percent beyond the

site’s optimum capacity can cost $100,000 in

lost revenue every hour.

Further complicating matters is the

huge unknown factor of mobile for 2010.

With the tremendous and ever-accelerating

proliferation of browser-equipped, app-loaded

smartphones, mobile is sure to play a bigger

role than ever, and likely to exceed even the

most optimistic retailer’s projections. All those

smartphones searching for products,

comparing prices, and making purchases

will be accessing the same back-end

databases and applications as all the PC-based

browsers, putting even more stress on

e-commerce systems.

THE FIRST BILLION-DOLLAR ONLINE DAY?One likely milestone in the upcoming 2010

holiday season will be the first billion-dollar

e-commerce day. Sales broke the $900

million mark on Green Tuesday (December 15

last year), and at least nine shopping days

exceeded $800 million.iv A major east coast

snow storm kept shoppers home-bound and

contributed to brisk online sales. This year,

mobile, with shoppers logging on even as they

are out in the stores, is likely to have an even

greater effect.

FIRST THINGS FIRST: UPDATESGetting a site into the near-final form it will

have for the holiday season is the first step to

making sure it will handle the demand. New

features and functionality — like enhanced

product presentation, reviews, and

Keynote Benchmark

This diagram illustrates actual results of a test of a major retailer’s Web site. The golden area illustrates that the more users who log on beyond the site’s optimum capacity, the more revenue is lost. A spike of 25 percent results in losses in the neighborhood of $100,000 every hour.

personalization options — should be well

under way if not complete by now. Sports

fashion leader Lacoste is one retail marketer

that understands the importance of readying

major site revisions early.

“We’ve been working on a huge

upgrade of our site and have been doing

side-by-side testing with Keynote in order to

measure the current experience against the

new experience,” Lacoste E-Commerce

Director Maryssa Miller told Benchmark.

“Our criteria is to be better in performance

than the current existing experience, even

with all the new functionality. And we want to

make sure that any kinks are worked out prior

to the busy holiday season, obviously.”

“We’re adding larger images with

greater resolution,” Miller continues. “We’re

adding new features to be able to check your

tax and shipping or your total costs much

earlier in the process. We’re also adding social

networking tools — the ability for people to

more easily share products through any of the

major social networks.”

BEST PRACTICE: BUILD PAGES LEANPerformance management starts with how the

pages are built, which often presents a

dilemma for retailers. With more and more

products approaching commodity status and

available at multiple online outlets, site

experience becomes a key differentiator.

Retailers want to create a rich experience for

visitors with interactivity, dramatic product

presentation, perhaps Flash, personalization

or other features to set themselves off from the

competition. But a heavy load of features and

functionality can drag site performance down,

often because of third-party content, and,

instead of making visitors sticky, can drive

them to leaner, faster competitive sites.

“Ideally, it’s best to focus on limiting

your number of third-party content and

domains on each page to roughly six domains

at most,” says Keynote Consulting Manager

Cliff Crocker. “As opposed to what we see for a

lot of retailers, where there’s 20 or 30 different

domains that are killing performance on

their page.”

Crocker also recommends keeping

the number of objects on a page to 40 or 50;

many retailers are packing their pages with

150 or more elements.

“You have to try to limit the time

that’s being spent in the browser,” Crocker

says. “The more JavaScript and the more

functionality a retailer adds to their page,

oftentimes can create very big delays within

the user’s browser.”

FACTOR MOBILE IN FROM THE STARTThe successful retailers this year will have

built mobile into their strategy right from the

start — not just as an afterthought to the

“main” site, but side-by-side with it. Shoppers

carrying smartphones are using them to check

prices, locate products, find deals, look at

reviews and, more and more, to make

purchases. (See “Shoppers, Start Your

Smartphones!” in Benchmark.) Many retailers

were surprised at the amount of mobile traffic

3

Keynote Benchmark

Lacoste is updating its site half a year ahead of the holiday season, and setting a performance bar even higher than last year. Bigger images and social media functionality are among the enhancements being implemented.

they got during the 2009 holiday season. And

there will be millions more smartphones in

the hands of shoppers this year.

“We were very surprised,” says

Lacoste’s Miller. “And I think that’s where we

really realized that we need to focus on

[mobile] for 2010. We already knew we

needed to focus on it, but I think [the 2009

season] was just really additional proof and

we knew that it needed to be a huge part of

our strategy for this year, and then continue to

be part of our strategy going forward.”

Because of the inherent slowness of

cellular networks and devices, mobile sites

need to be even leaner and meaner than wired

Web sites. It takes some hard decision-making

and analysis of what is essential for users

when they are browsing on the go and what it

takes to satisfy them, including their need for

speed. Search results can be confined to return

four or five results, for example, instead of the

40 or 50 that might be delivered on the wired

Web. And perhaps tracking pixels are needed

only on the landing page and cart page,

instead of every page on the site.

“There are some pretty strict rules

around how many elements you want to have

on a mobile page,” Crocker says. “Where we say

40 to 50 over the wired Web, we’re looking at 8

to 10 elements on a mobile page, just because

of all the latency that’s encountered over the

carrier networks. So step one is, you’ve got to

build a dedicated mobile site. There’s no way

you’re going to port what you’re putting up on

the wired Web or pare that down. You’ve really

got to build it from scratch.”

STEP 1: ESTABLISH YOUR BASELINE AND PROJECT TRAFFIC INCLUDING MOBILEKnowing your site’s baseline — how much

traffic it can handle right now — is the logical

starting point to begin understanding how

much work needs to be done. Then,

projections need to be made for how much

traffic and sales are expected for the holiday

season. Typically the purview of the sales and

marketing team, this is where it gets a little

tricky; consumer behavior is difficult to

project, especially in these times of continued

economic turbulence. Whatever the traffic and

sales projections, it’s up to IT and the Web

team to make sure the site can handle any

unexpected surge in site visitors.

“From the technology side, you

should always treat whatever the business tells

you with a grain of salt,” says Keynote Director

of Global Testing Services Donald Foss. “You

may say, ‘I’m going to add 50 percent as

planning-appropriate, and then I’m going to

add another 30 or 50 percent as a technology

buffer — an insurance policy,’ and so it ends

up roughly double, or maybe 250 percent, of

what the marketing department projects.”

Many, if not most, retailers still fail

to account for mobile in their traffic

projections, and that could be a bigger mistake

than ever this year. Visitors logging on from

mobile devices are calling on the same

resources and databases as visitors coming

through PC browsers, and so are adding to

overall site load. Making matters worse is that

those mobile connections are extremely slow;

traffic coming in through mobile actually holds

the connections longer, and consumes more

than its share of resources.

“In planning for traffic, it’s important

to account for the total sum of the traffic from

a holistic standpoint,” Foss says. “It’s

important to make sure you’re actually testing

the mobile site and the regular Web traffic all

at the same time, in the correct proportions

and in the right demographics, to see how

it works.

“Early on, you take a multi-pronged

approach. You test the core Web site by itself.

You test the mobile site by itself, and make sure

they’re both tuned and working correctly. Then

you start the holistic testing, with the full sum of

traffic from the correct sources that you predict.

If 15 to 20 percent of user traffic is coming over

mobile, then we will actually have 15 to 20

percent of the test traffic running mobile-type

connections, and make mobile scripts with

mobile line speed coming into the site.”

For some retail marketers such as

Lacoste, the mobile and wired Web

experiences are tied tightly together, so the

necessity for holistic testing is apparent.

Speaking about their iPhone app,

Lacoste’s Maryssa Miller says, “My favorite

part is that it’s actually connected to the

e-commerce site. I’ve seen other apps where it

is a very separate experience, but ours is very

much integrated, which I think is great

because we’ve found that a lot of people will

save items in their shopping cart and, for

whatever reason, they don’t purchase them.

This way, if they decide they want it, they can

access their shopping cart on the e-commerce

site or directly from the app. That’s a really

usable feature and one of the best things

about it.”

MAKE TESTING A REAL REALITY CHECKIn a very oversimplified nutshell, with load

testing you are throwing users at the site until

it breaks (which is why live-site load testing is

typically done in off-hours). But to be realistic,

accurate, and reliable, a load test has to be far

more sophisticated than simply simulating a

gross volume of visitors showing up, which is

the essence of the often-used but imperfect

“concurrent user” methodology.

A concurrent user approach delivers

a picture of performance that is far from

accurate. Basically, it measures a static

threshold. Here’s 100 thousand users. Does it

break? OK, add another 10 thousand. And

another. The methodology assumes incoming

users will patiently wait their turn to enter,

and that everything will be fine until they do.

But in reality, those additional users are trying

to log on and being turned away, and the

experience for the users already on the site

degrades, often quickly. Testing only for

concurrent users does not accurately mirror

user behaviors and actual site traffic dynamics.

4

Keynote Benchmark

In reality, users are continuously coming and going. Those with slow connections or mobile devices are taking longer. Some who are already familiar with the site are in and out quickly. When the site slows down under the load, more and more users pile on, because it’s taking longer for the previous users to complete their tasks and leave. The load is never static. And users are not merely numbers going in and out. They are individuals with individual thresholds of patience — and that is the critical factor to know when it comes to converting visitors to sales, and minimizing revenue lost due to poor performance. A more accurate representation of site load is created using an “arrival rate”

methodology, which factors the peaks and valleys of traffic and availability based on likely user behaviors and site slowdowns. Hand-in-hand with this methodology is behavior modeling, which factors, among other things, users who are familiar/unfamiliar with the site, their tolerance for delays, and their tenacity in sticking with the site until they accomplish their tasks. “There’s a certain amount of slowness that can be tolerated, and a certain amount that won’t be,” says Foss. “That’s latency tolerance. There’s also tenacity, which means essentially, how important is it for a person to finish the transaction they’re on? How likely are they to continue with it or not?” If they’re checking out, they’re more likely to be patient with order and credit card processing. But if they’re elsewhere on the site, tenacity may be significantly lower.

BEST PRACTICES: METHODOLOGY & MODELINGTo really understand how your site

performance will hold up — or not — under

holiday stress, and to understand what the

experience will be like for users, use an arrival

rate methodology and factor in behavior

models for the many, many types of users and

tasks your site will serve.

Behavior modeling results in

numerous permutations (often thousands)

combining these variables:

• Familiarity: experienced users vs.

newcomers

• Connection speed: super-fast FIOS vs.

super-slow mobile device, and everything in

between

• Latency tolerance: patience of users with

slow site response

5

Keynote Benchmark

Lacoste’s iPhone app is tied directly to their e-commerce site, so registered users can access their cart and make purchases from either place. With mobile sites typically calling on the same backend resources as the wired site, it is important to test the two simultaneously to evaluate overall load-handling capability.

The “concurrent user” model assumes a constant volume of site traffic, which is a poor representation of the ups and downs of actual traffic. Arrival rate methodology, on the other hand, combined with user behavior modeling, mirrors much more closely the patterns and volumes of peak site traffic.

• Interaction speed: complexity of the page to

navigate, and attention level of the user

• Tenacity: willingness of users to stick with a

task through completion

“We’re not just replaying a

transaction multiple times ad nauseam to try

to put load on the site,” Foss says of Keynote’s

LoadPro testing. “There’s a whole lot more

going on behind the scenes. There’s literally

hundreds or thousands of different types of

users that we emulate during a single test to

make sure we’re emulating what a real user

population would be.”

TEST IN THE REAL WORLD — ALL OF ITTo know how your site will perform for users

dispersed across the country or the world,

testing must be done over the Internet, from

the same geographic locations as your users,

not from behind the firewall. There’s simply

no way to simulate the vagaries of Internet

backbones, third-party content feeds, CDN

performance, and signal transmission through

the critical last mile — unless you are at the

end of that mile, with a browser.

With testing agents dispersed where

your users are, you get an accurate picture of

variations in performance, and overcome the

danger of looking at averages. An average

page-load time of three or four seconds may

seem OK, but that kind of average could mean

your page is loading in one second for

someone in New York, but taking six or more

seconds for someone in Chicago. And that is

not likely to be acceptable. The solution is to

test from multiple, geographically dispersed

locations, look at the data, and address any

local or regional bottlenecks.

STAFF UP AND STOCK UP — ON CAFFEINE AND CAPACITYRetailers who put off load testing can look

forward to many sleepless nights in October

and early November as they try to get their

sites up to speed. But even the best-in-class

retailers can count on a sleepless night or

two — particularly Thanksgiving night — as

they stay up to make sure all their hard work

has paid off.

“One of my biggest clients spent the

night at headquarters on Thanksgiving night

last year,” Foss says. “He’s up all night making

sure everything is running perfectly, and goes

home Black Friday morning at 9:00 a.m.”

The holiday shopping season is the

culmination of many hard hours of work for

the IT/Web department. And no matter how

well things are planned, no matter how

rigorously everything is tested, there’s always

the chance that the unexpected will happen

and something will go wrong. So it makes

good sense to have technical personnel on

hand and on call during all the critical

shopping periods to handle any emergencies,

and to have extra computing capacity standing

by just in case it’s needed.

DON’T WAIT ANOTHER MINUTEIf you haven’t started load testing your site

yet, there’s no time to lose. “The most

successful retailers look at holiday readiness

as an ongoing process,” says Crocker. “They

start in January and they test and they

monitor performance and they benchmark

and they trend all the way through until the

end of December.”

And December is just a few short

months away.

6

Keynote Benchmark

.............................................................................................................................................................................© 2010 Keynote Systems, Inc. All rights reserved. The trademarks of Keynote Systems, Inc. include Keynote®, DataPulse®, CustomerScope®, Keynote CE Rankings®, Keynote Customer Experience Rankings®, Perspective®, Keynote Red Alert®, Keynote Traffic Perspective®, Keynote WebEffective®, The Internet Performance Authority®, MyKeynote®, SIGOS®, SITE®, keynote™, The Mobile & Internet Performance Authority™ and all related trademarks, trade names, logos, characters, design and trade dress are trademarks or registered trademarks of Keynote Systems, Inc. in the United States and other countries and may not be used without written permission.

ENDNOTES

i. U.S. Department of Commerce

ii. comScore press release, “comScore Reports $29.1 Billion in U.S. Retail E-Commerce Spending for Full November-December

Holiday Season, UP 4 Percent vs. Year Ago, 1/6/2010

iii. Internet Retailer, “Total holiday retail sales rise 1%, trailing e-retail’s 4% increase

iv. op cit