Northway Solutions Group 9005 Overlook Blvd • Brentwood, TN. 37027 • Ph: 866.611.8762 • Fax: 866.611.8762

www.northwaysolutions.com

Testing Authorities

Process and Vision

Testing Authorities 2

The CoE Maturity Model

A testing center of excellence (CoE) is an entity with centralized test technology and

methodology that helps drive quality and performance initiatives across the enterprise. What is

the highest level of maturity in a Testing Center of Excellence? HP describes it as an Authority.

The Maturity Model for a Center of Excellence as it applies to Quality and Performance was

established as a guideline for companies in a whitepaper titled “Application Delivery: Center of

Excellence Evolution” by Mercury Interactive in 2004. It describes the move from a Service

Utility to a Quality and Performance Authority as one of “institutionalizing tools, techniques,

and practices”. This is a general roadmap for a Center of Excellence. Below is an example of the

maturity model according to HP Software as it looks today, as HP continues this vision:

Testing Authorities 3

In the ultimate achievement of this model (aka “Optimized” or “Authority”), the following

characteristics are true:

The CoE has proven time and again that the process is effective in mitigating as much

risk as possible associated with production roll out.

The success of the CoE is marketed to the business regularly, and has become

recognized as a critical success factor for projects.

Year over year, metrics consistently demonstrate that the processes within the CoE are

the best approach for quality and performance testing initiatives.

Projects that operate outside the CoE process exhibit problems that are exposed quickly

and are highly visible to the organization.

To get such metrics requires traceability, real-time visibility, and consistent “marketing” style

evangelism into all areas of IT to obtain “buy in”. The concept and value of a CoE is a hard sell at

first, and every day is a battle; however, for those willing to go the distance there is a vast

reward. Management dreams about continuous improvement, efficiency, and return-on-

investment. These aspirations can be realized with a properly implemented Center of

Excellence.

For years this maturity model has been presented at Enterprise software conferences where

the topics are focused on quality and performance of applications. However, reaching the

Authority level of maturity is more than just deciding to do it. Most of the time, there is no

discussion of the challenges that teams will face when moving from a Service Utility to an

Authority.

Hint: It requires a long standing, solid track record of showing value. Organizations should not

expect to get there very quickly, or without an investment of people and budget.

Testing Authorities 4

What Does it Take To Get There?

What does maturity look like? What are the requirements and changes that must be made to

progress to the highest level of maturity?

It requires a “charge back” or shared service model where all the Lines of Business (LOB)

contribute to the budget of the Performance team in order to fund the resources and

the hours allocated for their projects. Some companies cannot move to this model

because of the way budgeting is set up. The testing team is seen as a service that must

be able to handle whatever is asked of them. Under those conditions, a charge back

mentality should be in place, and worked into the escalation model to management. For

example, when the CoE is requested to engage a new project, instead of the project

allocating specific budget to fund a resource, a well-defined process for upper

management approval needs to be in place that establishes the use of CoE resources. If

too many projects are started simultaneously, it is management's responsibility to

allocate more of the budget for third party contractors to handle overflow, to fund

additional FTE resources, or to prioritize projects and push them later on the

implementation calendar. With enough demands made from the business side, this

usually causes management to spend an inordinate amount of time with the approval

process, and the charge back model becomes a lot more attractive at that point. Few

companies ever become an Authority when they do not have a good charge back

model/mentality.

It requires a culture of sharing knowledge, and not hoarding it. This means a testing

“community” (not just made of testers, but all people who are on board to make Quality

and Performance their personal responsibility). The testing community should be

sharing ideas and advice across the different technology and business groups, and

collaborating on a regular basis for joint-success. The impact of the loss of any individual

team member is mitigated, because the knowledge level and intellectual capital remains

within the community.

Testing Authorities 5

A CoE operating as an Authority is typically made up of team members that have become

recognized as industry experts, and are known outside the organization and sought after for

their knowledge. These resources frequently speak at user groups, conferences, and other

industry events. The skill sets are so wide and deep in the group that outside consultants rarely

bring anything new to the group, and are only used to complete overflow work during sudden

spikes in activity. Internal training on the products and process rival that of external formal

training offered by the vendors themselves.

The CoE Authority has proven by a history of successes, which demonstrate that their

centralized model, processes, and techniques have positively affected the IT, and upper level

management has officially recognized their contributions. In short, they have earned the right

to be an Authority.

What Needs To Be Done

How do you reach this pinnacle of software testing? What are the things that need to happen

to begin acting as an Authority?

The first recommendation would be to set up a “Standards and Methodologies” steering

committee, made up of experts and key players in the decision making process for IT. This

would include QA and Performance managers, members of the business management team,

the CTO, etc. This group is responsible for quality and performance engineering, standards

enforcement, and instituting formalized process improvement. A balance between rich process

and the level of process that is practical needs to be established first. There should be enough

flexibility to meet the needs of all projects and lines of businesses. More flexibility should be

allowed for projects that feature a lower amount of risk to the business if they fail, and less

flexibility to those who are mission critical.

It can be difficult in many organizations to get participation in the development of standards

and methodologies. If that is an issue, an alternative might be setting up two sets of teams. The

first would be a steering committee made up of the decision makers who will be needed

Testing Authorities 6

support a successful implementation. The second is the group of leaders who make the

decisions and do the actual implementation. This “working group” is a team of “experts” that

can define the processes and standards, and work though all of the issues around breaking

down the fiefdoms.

The Standards and Methodologies committee must have the final say in allowing an application

to be released into the production environment. If it doesn’t meet the standards, it doesn’t go

to production. If the product managers or the business are allowed to override this or go

through any “back door” channels, it is indicative that there is no real “Authority” in place. For

ITIL shops, this might be a change advisory board (CAB). The Working Group mentioned before

would develop the standards and methodologies and focus on building the CoE. The CoE would

have a representative that sits on the CAB and has the authority to hold up a release. There

might even be a special meeting to review performance testing results prior to sending it to the

CAB for approval.

Ultimately the BUSINESS has to grant this right, not the CIO. The people who own the business,

the board of directors, and the shareholders need to be partakers in this decision. Obviously

this is easier said than done. The danger of empowering the Authority is that too much process

can bog down the software development lifecycle and create a bloated, endless maze of rules

that people will always try to work around instead of championing.

Create documentation (using a technical writer) that describes the process for meeting the

requirements set by the Authority, and how to comply with an audit of that process.

Audit the lines of business. This means that there has to be a penalty for non-compliance.

Penalties might include denying access to the testing process until the line of business or

department complies, denial of production implementation, compensation penalties (bonus or

raise), and negative marks on annual performance reviews for the management resources over

the project.

Testing Authorities 7

Remove the authority of project teams in the individual lines of business to bypass the process.

They cannot continue do it their own way, or cut corners. This will be a huge challenge in places

where the lines of business do not wish to give up that level of control (i.e., those who may be

from a past merger acquisition who still operate independently).

There needs to be a high level of visibility to the C-Level and to the business through a

dashboard or other reporting mechanism that allows them to keep track of the progress, look

at metrics, wins, and losses. Each year this information should be used to determine how well

the Authority is progressing. If the COE ceases to deliver what it has promised to the business

with the authority given, then the authority should be taken away.

Process and Penalties

Obviously, these things are much easier to write about than actually do for most companies in

today’s IT culture. This is not just a check list of items that can be completed in a couple of

months. The key word is “maturity”. When a company’s web site states they are CMM Level 5

Certified, and they are less than six months old, something isn’t right. To trust that company for

their product or service is the equivalent of handing a two year old child a machine gun and

letting them go play in the yard. Maturity comes by making mistakes, learning from those

mistakes, and making improvements over time that allow easy navigation over all the road

blocks and sand traps that are out there.

Some have made the argument for an internal “contract” between IT and the business to exist.

This represents a formal agreement that lays out the terms and conditions for meeting Quality

and Performance standards for applications, with penalties on both sides when the terms are

not met. If IT breaks the contract, then CIO’s and CTO’s are fired, bonuses are denied, and

managers of projects who systematically defeated the agreed upon process lose their job. If the

business breaks the contract (by skirting the processes by constantly making exceptions), then

the cost immediately skyrockets because of the effort required for doing things over when

failure occurs – thus affecting their budget. In essence, IT is saying, “To do it right is 10 dollars,

to subvert the standard its 40 dollars, paid upfront and guaranteed”.

Testing Authorities 8

Unfortunately, the concept of a formal contract is not based in reality. Businesses do not have

to remain in business and can make bad decisions. Sometimes business decisions are made that

are bad for IT because, at that time, there were other things that were more important to the

business than process. A better idea than a contract with penalties is a formal mechanism for

overriding the process. Informal mechanisms already exist in most companies – when Bob

decides he doesn’t want to do things the right way, he just does it his way. If there is a CoE in

place, this mechanism should be formalized, and there should be guidelines for using it. This

mechanism needs to exist, and be enabled when it is needed. There also needs to be metrics to

determine the cost/impact using the overriding mechanism – bringing it back to real dollars if

possible. The Steering Committee within a company might be able to report back to the

shareholders that, “… because process standards were circumvented, a five million dollar

project ended up costing ten million. This caused an increase in support for the application by

150%”. CIO’s who understand this can build exceptions into the annual budget, by planning on

a certain percentage of projects to bypass the process. If a dashboard or report is available that

can show where and when the Quality and Performance process was bypassed and capture the

extra cost, then a pattern of behavior can be documented, which shows the activities that cost

the company more money than necessary and can expose what could be construed as a career

limiting move.

As companies set their sites on the all-elusive Authority, they need to be prepared for the long

haul. It can be achieved, and just like with any worthy improvement exercise, it won’t

necessarily be the most pleasant experience. Working toward that maturity will ensure that

across all tiers in the organization, and across all areas of the software lifecycle, that sound

practices and process are completely embedded and permeated into all layers of the IT realm.

At that point, congratulations are in order. For many years, this idea of an Authority has been

touted as achievable, but few choose to take the hard road and go down the path.

Testing Authorities 9

The Virtual Center of Excellence™

Is there an alternative way to shorten the time it takes to reach an Authority? Absolutely! Some

have been brave enough to go before you into uncharted territory and have cleared out a path

finding out what works and what does not. As a result, they can share best practices and

standardized processes that have proven to be successful in a variety of industries. Northway

Solutions Group has experts across multiple business verticals that have the experience and

know how to customize a process for a specific need. In short, we’ve built Testing Centers of

Excellence internally, which we call the Virtual Center of Excellence™ (vCOE). Using our team of

consultants, the time to maturity can be cut exponentially. Here’s how it works:

Testing Authorities 10

The client engages the vCOE to provide services in one of the following centers:

Performance Testing Automation

Functional Testing Automation

Specialty Testing (includes SOA testing and Service Virtualization)

We provide the senior level expertise and testing services to the client, completing projects by

leveraging the Northway Virtual Team (the “vTeam”). The vCOE acts as an extension of the

client’s own testing organization, providing the client with all project artifacts and deliverables.

This is NOT traditional outsourcing! The vCOE provides the same level of skill sets as those

found in onsite; niche player consultants who hit the ground running day one of a project,

working strategically through all of the phases of the testing project. Although the pricing

model is very competitive, the vCOE doesn’t try to compete with offshore providers on price,

but rather on skill sets, which ultimately impacts timelines. This is especially helpful when a

company is initially forming QA and performance teams, and may need guidance from a more

mature organization that already has best practices in place.

Testing Authorities 11

Many companies have already invested in building out a COE of their own, or are in various

stages of maturity. Other companies do not wish to invest the capital to build out a COE within

their company structure, but want an outsourced solution that adds value on day one. No

matter what your company’s situation, Northway has designed a strategy to work within your

organization. There are currently three defined delivery models:

vCOE™ Delivery: A fully virtualized Center of Excellence - delivering complete testing

projects on behalf of the client as the single source for COE services.

COE Expansion: Leverage the vCOE™ to extend a client’s own internal COE, providing

additional capacity or specialized skills.

COE Formation: Leverage the vCOE™ to execute on behalf of a client while the client

builds out their own internal COE capabilities.

Bringing in an experienced strategic consulting organization to help build out the COE can

reduce maturity time by 50 to 70 percent. The vCOE™ allows companies to get started in just

days by “bolting on” a team of experienced resources who can take on complex projects and

who already have a repeatable methodology to handle the testing cycle. This is different than

trying to reach a 36-month journey in six months with what you know today. It is about

allowing someone who has made the journey lead you around all the landmines and obstacles

in your way because they have already made the journey before you.

This approach is not for every company, especially those who only need typical, short term

consulting engagements with onsite resources. It works better for those who are currently

overwhelmed with the number of projects they have and are constantly getting requests for

new projects, and have serious issues hiring qualified resources fast enough to handle them. It’s

also a better fit for those who may have experienced pains with offshore resource efficiency,

competency, communication barriers, or the higher costs of global software licensing models.

Whatever direction you take, the path to maturity is a challenge even for the most experienced

IT professionals. Take steps to begin your journey today.

Testing Authorities 12

About Northway Solutions Group

Northway Solutions Group is a technology solutions provider that provides consulting services

around HP's IT Performance Suite of products. As an HP Elite/Specialist software partner, we

specialize in the sales, implementation, education and support of HP's Application Lifecycle

Management (ALM) platform, which includes Quality Center, Unified Functional Testing,

Performance Center, LoadRunner, and Service Virtualization. Northway employs only the most

qualified certified consultants and instructors with real-world experience and who can provide

long-term solutions to the toughest business challenges. Visit www.northwaysolutions.com to

find out more information about our services and offerings.

About HP

HP is a technology solutions provider to consumers, businesses and institutions. HP software

helps IT organizations make sure that every dollar delivers positive business outcomes. Their IT

Performance Suite of products automate key processes across IT functions, and their OpenCall

platforms enable people to communicate and access services. As the focus of IT shifts from

systems management to business results, companies are investing in HP software to lower

costs, improve delivery time and reduce risk.