The WV Development Office Has Not Improved Performance Measures

8/8/2019 The WV Development Office Has Not Improved Performance Measures

1/25

AGENCY REVIEW

DEPARTMENT OF COMMERCEDEVELOPMENT OFFICE

August 2010

PE 10-06-473

AUDIT OVERVIEW

The Development Ofce Has Not ImprovedPerformance Measures

The Development Ofce Has Improved ItsEmployee Evaluation System

WEST VIRGINIA LEGISLATIVE AUDITOR

PERFORMANCE EVALUATION & RESEARCH DIVISION


2/25

JOINT COMMITTEE ON GOVERNMENT OPERATIONS

Senate

Edwin J. Bowman, Chair

Herb Snyder, Vice-Chair

Walt Helmick

Brooks McCabeClark S. Barnes

House o Delegates

Jim Morgan, Chair

Dale Stephens, Vice-Chair

Sam Argento

Ruth RowanPatti Schoen

Craig Blair, Nonvoting

Scott G. Varner, Nonvoting

Agency/ Citizen Members

Dwight Calhoun

John A. Caneld

W. Joseph McCoy

Kenneth QueenJames Willison

JOINT COMMITTEE ON GOVERNMENT ORGANIZATION

Senate

Edwin J. Bowman, Chair

Herb Snyder, Vice-ChairRichard Browning

Dan Foster

Jerey V. Kessler

Brooks McCabe

Joseph M. Minard

Corey L. Palumbo

Randy White

Bob Williams

Jack Yost

Donna J. Boley

John Shott

Dave Sypolt

House o Delegates

Jim Morgan, Chair

Dale Stephens, Vice-ChairSam J. Argento

Brent Boggs

Greg Butcher

Samuel J. Cann, Sr.

Roy Givens

Daniel J. Hall

William G. Hartman

Barbara Hateld

Mike Manypenny

Dale Martin

Daniel Poling

Meshea L. Poore

Margaret A. Staggers

Randy SwartzmillerJoe Talbott

Terry Walker

Tom Azinger

Daryl E. Cowles

Pat McGeehan

Carol Miller

Jonathan Miller

Thomas Porter

Ruth Rowan

Aaron Allred

Legislative Auditor

Building 1, Room W-314

State Capitol Complex

Charleston, West Virginia 25305

(304) 347-4890

John Sylvia

Director Research Manager

Tina L.C. Baker

Research Analyst



Reerencer

Brian Armentrout Derek Thomas


3/25

Performance Evaluation & Research Division | pg.

Agency Review August 2010

CONTENTS

Executive Summary .................... ................... .................... .................... ................... .................... .................... ................... .......... 5

Objective, Scope and Methodology .................. .................... .................... ................... .................... .................... ................... 7

Issue 1: The Development Ofce Has Not Improved Perormance Measures ................... .................... ................... 9Issue 2: The Development Ofce Has Improved Its Employee Evaluation System ................. .................... ..........15

List O Tables Table 1: Levels o Compliance .................. ................... .................... .................... ................... .................... .................... ......... 5

List O AppendicesAppendix A: Transmittal Letter to Agency ..........................................................................................................................19Appendix B: 2009 Trade Show Cost Analysis ......................................................................................................................21Appendix C: Agency Response ...............................................................................................................................................23


4/25

pg. | West Virginia Legislative Auditor

Development Ofce


5/25



EXECUTIVE SUMMARY

This review is an update of the September 2007 PerformanceEvaluation of the West Virginia Development Ofce (WVDO). Thepurpose of this update is to determine whether WVDO has complied with

recommendations made in that report. Of the six recommendations withwhich compliance was evaluated, the WVDO demonstrated compliancewith one, partial compliance with three, and non-compliance with tworecommendation.

In response a recommendation made in the previous report,the Executive Director issued a directive requiring annual employeeevaluations. The WVDO has also incorporated outcome measures inaddition to input measures into its evaluation system and demonstratesplanned compliance with the recommendation that evaluation forms befully completed. Although the WVDO uses multi-state comparisons todetermine West Virginias advantages over other states in responding to

preliminary contacts, no efforts are made to track the results of these preliminary contacts in order to determine which methods are moreeffective at attracting businesses. The WVDO has not complied withthe recommendation to develop outcome measures for determining theorganizations performance.

This update uses the following designations for compliance withprevious recommendations made by the Legislative Auditor:

Table 1

Levels of Compliance

In Compliance The Division has corrected the problem(s) identied in theprevious report.

Partial ComplianceThe Division has partially corrected the problem(s) identied

in the previous report.

Planned Compliance

The Division has not corrected the problem(s) identied

in the previous report, but has provided sufcient

documentation that the agency will do so in the future.

In DisputeThe Division does not agree with either the problem

identied or the proposed solution.

Non-Compliance The Division has not corrected the problem(s) identied inthe previous report.

Requires Legislative Action

The recommendation was intended to call the attention of

the Legislature to one or more issues that may or may not

require statutory changes.

Legislation EnactedThe Legislature took legislative action to issues raised by the

Legislative Auditor in the 2007 report.

O the six recommendations withwhich compliance was evaluatedthe WVDO demonstrated compliance with one, partial compliance with three, and non-compliance with two recommendation


6/25


Development Ofce


7/25



OBJECTIVE, SCOPE & METHODOLOGY

Objective

This agency review of the West Virginia Development Ofce

(WVDO) was conducted as part of the Departmental Review of theWest Virginia Department of Commerce mandated by West VirginiaCode 4-10-8(b)(3). The objective of this review was to determine theWVDOs compliance with recommendations made in a 2007 report bythe Legislative Auditor

Scope

The scope of this report was actions taken by the WVDO sincethe release of the Legislative Auditors report in September 2007.

Methodology

In order to determine the WVDOs compliance with previousrecommendations, the Legislative Auditor corresponded with WVDOstaff and reviewed information found in the West Virginia Code of StateRules, WVDO reports and other publications, and employee evaluationforms.


8/25


Development Ofce


9/25



ISSUE 1

Since outcome measures arelinked to the agencys missionthese are more indicative o thesuccess and eectiveness othe agencys programs. Consequently, the Legislative Auditoindicated in the 2007 report thaWVDO should develop outcomemeasures or all its programs.

The Development Ofce Has Not Improved PerformanceMeasures.

Recommendation 1

TheDevelopmentOfceshoulddevelopperformancemeasuresthatmeasuretheoutcomesofitsprograms.

Level of Compliance: Non-Compliance

An organizations performance is measured by several interrelatedelements:

Inputs - the amount of a particular resource, such as money orstaff time, used to produce results;

Activities - the actions taken by a program to produce results;

Outputs - the amounts of products or services produced, such asclients served or business contacts made;

Outcomes - changes, such as an increased number of foreigninvestors, indicating progress toward the organizations missionor objectives; and

Efciency measures - a cost per unit measure indicating how

efciently the organization is producing results.

In the September 2007 evaluation of the West Virginia DevelopmentOfce (WVDO), the Legislative Auditor found that the WVDO was notutilizing outcome measures to determine the effectiveness of its programs.Instead, it was relying on input, output, and activity measures in all buttwo of its divisions. Since outcome measures are linked to the agencysmission, these are more indicative of the success and effectiveness of theagencys programs. Consequently, the Legislative Auditor indicated inthe 2007 report that WVDO should develop outcome measures for allits programs. When asked whether outcome measures had since beendeveloped, the WVDO pointed to the following:

Evaluation of trade shows by the WVDO beginning in 2008;

WVDO support of initiatives such as the reform of WorkersCompensation, phased reduction of the corporate net incometax rate, and phased reduction and eventual elimination of thebusiness franchise tax; and


10/25

pg. 10 | West Virginia Legislative Auditor

Development Ofce

The evaluation o trade shows ascurrently conducted by WVDO isnot a measure o outcomes, buta measure o inputs (trade showcost) and outputs (number oleads, contacts, or suspects).

The use of multi-state comparisons in preparing and analyzingeconomic development marketing materials.

The 2009 evaluation of trade shows is provided in Appendix B.As the appendix shows, in 2009 the WVDO measured the cost of eachtrade show; the number of leads, suspects, and contacts made at eachshow; and the cost per lead, suspect, and contact. For the 2010 analysis,the term suspect has been eliminated due to a strong similarity betweensuspects and leads. Both suspects and leads refer to companies thatillustrate the potential for investment projects, but more information isknown about leads. As used in this analysis, the WVDO denes contactsand leads as follows:

Contactaqualitycompanyinoneofourtargetsectorsthatmeritscontinuedfollow-

up but shows no immediate investmentopportunity.

Leadacompanythathasexpressedplansfor an investment project in the shortorlongterm.

Both contacts and leads are generatedfrom a business or industry; not tradeorganizations.

The WVDO indicated that trade show evaluation illustratesthecost

perleadversustheactualqualityoftheleadobtained;therebyallowingustomaximizeoutcomesinacosteffectivemanner. Additionally, theWVDO stated that the calculations were preliminary and that variableswould not be seen until year ve of evaluation. When questioned by theLegislative Auditor as to what variables would be included in the long-term analysis, the WVDO indicated the continued use of the formulashown in Appendix B; e.g., the number of contacts/leads generated andassociated costs of a trade show. The evaluation of trade shows ascurrently conducted by WVDO is not a measure of outcomes, but ameasure of inputs (trade show cost) and outputs (number of leads,contacts, or suspects). In order to determine the effectiveness of tradeshows, the WVDO should expand the scope of the evaluation process

to track the outcome of leads and contacts made how many leads orcontacts actually result in the establishment of investment projects.

The support of initiatives by the WVDO such as the reform ofWorkers Compensation, phased reduction of the corporate net incometax rate, and phased reduction and eventual elimination of the businessfranchise tax is also not an outcome measure, but an activity measure.The Legislative Auditor inquired whether there are plans to track the

The support o initiatives by theWVDO such as the reorm oWorkers Compensation, phasedreduction o the corporate net

income tax rate, and phased re-duction and eventual eliminationo the business ranchise tax isalso not an outcome measure,but an activity measure.


11/25

Performance Evaluation & Research Division | pg. 1


The use o multi-state compasons in preparing and analyzingeconomic development marketing materials is also an activitymeasure, rather than an outcomemeasure.

resulting outcomes of these initiatives in order to determine the impact onbusiness development in West Virginia, to which the WVDO responded:

tracking outcomes of any specicbusinessinitiativewouldbeunreliableandpotentiallymisleading(nottomentionaninefcientuseofscarceresources)duetothefactthatbusinessdevelopmentinWestVirginiadependsuponavarietyoffactors.However it is clear that aggressive taxmodernization efforts, quality trainingprograms,innovativereformssuchastheprivatization of workers compensation,andtheimplementationofboldinitiativessuch as the High Technology Business

Property Valuation Act and AircraftValuation Tax Credit help to create apositive business climate andare knownvariablesinpromotingbusinessgrowth.

The use of multi-state comparisons in preparing and analyzingeconomic development marketing materials is also an activity measure,rather than an outcome measure. Regarding this comparative data, theWVDO states the following:

The WVDO continuously updatescomparative data it uses daily when

preparing proposals for prospectivebusinessesandtorespondtorequestfromcompanies, local economic developmentauthorities, and site location consultantsfor information. Inorder toattract andexpand businesses, it is essential tounderstand the factors that businessesconsider when theymake locational andexpansiondecisions.TheWVDOanalyzesthesefactorsandpromotesthosewherewehaveadvantages.

The WVDO did indicate that it is currently in the process ofdeveloping a Customer Relationship Management system, stating thefollowing:

TheWVDOisworkingwiththeOfceofTechnologytodenetheneedsacrosstheagency in order to develop a CustomerRelationshipManagement (CRM) system

The WVDO is currently in tprocess o developing a Customer Relationship Managemen

system.


12/25


Development Ofce

The development o the CRM

system will provide the WVDOwith an opportunity to establishoutcome measures in compli-ance with the prior recommen-dation made by the LegislativeAuditor.

which will provide management withthe appropriate tools to manage andmore efciently determine statewide

investments and their impacts on thestateThe proposed CRM system willprovidetheWVDOwithacomprehensivedatabasewhichwillassimilatedatafromthe various divisions within the ofce;therebyallowingmanagementtoaccessabroadrangeofinformationtobeusedfortracking,reporting,analysis,etc.Atthistime,weareintheearlystagesofanalysisandtimelinesarebeingestablished

The development of the CRM system will provide the WVDO

with an opportunity to establish outcome measures in compliance withthe prior recommendation made by the Legislative Auditor. Developmentof outcome measures will enable the WVDO to effectively gauge theimpact of its programs in order to better meet the needs of the state andfulll its mission.

Recommendation 2

TheDevelopmentOfceshouldconsiderreviewingandanalyzingrecordsofpreliminarybusinesscontactstoassistinidentifyingpractices

thataremoreeffectiveinattractingbusinessinvestmenttothestate.

Level of Compliance: Partial Compliance

In the September 2007 evaluation of the WVDO, the LegislativeAuditor found that the WVDO was not reviewing records of preliminarycontacts with businesses and recommended that the WVDO begin doingso, stating the following:

Organizing and analyzing records ofpreliminarycontactactivitywithbusinessesshowing interest inWestVirginiacanbe

valuable in identifyingpractices that aremore effectiveatattractingbusinesses tothestate.

The WVDO indicates compliance with this recommendationthrough the use of the multi-state comparative data to prepare proposalsfor and respond to information requests from businesses, developmentauthorities, and location consultants. However, when asked whetherrecords of these contacts are maintained in order to track the requests

The WVDO uses multi-statecomparative data to prepareproposals or and respondto inormation requests rombusinesses, developmentauthorities, and locationconsultants. However, recordso these contacts are notmaintained in order to provideinsight into what methodsattract or ail to attract businessinvestment to the state, whichwas the intent o the LegislativeAuditors recommendation.

In the September 2007 evalua-tion o the WVDO, the LegislativeAuditor ound that the WVDOwas not reviewing records opreliminary contacts with busi-nesses and recommended thatthe WVDO begin doing so.


13/25



The WVDO indicated that it hanot consulted with any outsideentity to develop outcome mea

sures.

made and the resulting outcomes, such as whether a business locates inWest Virginia after receiving proposals or information, the WVDO statesthe following:

tracking outcomes of any specicbusiness initiative would be unreliableandpotentiallymisleading(nottomentionaninefcientuseofscarceresources)dueto the fact that business development inWest Virginia dependsupon a variety offactors.

The ability to compare West Virginias advantages anddisadvantages to those of other states in preparing information andproposals for entities is certainly an important tool. However this does

not provide insight into what methods attract or fail to attract businessinvestment to the state, which was the intent of the Legislative Auditorsrecommendation.

Recommendation 3

TheDevelopmentOfce,oranotherqualiedentitysuchastheWest VirginiaUniversity orMarshallUniversity SchoolsofBusiness,should develop outcome measures internally to fairly and accuratelyreporttheresultsofagencysefforts.

Level of Compliance: Non-Compliance

In response to the lack of outcome measures provided by theWVDO during the 2007 evaluation, the Legislative Auditor conducted ameasure of outcomes by calculating per capita Foreign Direct Investmentand export dollars for West Virginia and competitor states in order togauge performance of the International Division of the WVDO. Itwas recognized that these measures might not be the most meaningfulin determining the states economic competiveness. Therefore, it wasrecommended that the WVDO develop outcome measures to accuratelygauge the results of agency efforts or that the WVDO utilize the expertise

of another qualied entity, such as the West Virginia University orMarshall University Schools of Business to do so. As discussed underRecommendation 1 above, outcome measures have not been developedby the WVDO. Upon inquiry from the Legislative Auditor, the WVDOindicated that it has not consulted with any outside entity to developoutcome measures.


14/25


Development Ofce


15/25



In the 2007 evaluation, the Legislative Auditor recommended thathe Division conduct evaluationsannually.

The Development Ofce Has Improved Its EmployeeEvaluation System.

Recommendation 4

TheLegislativeAuditorrecommendsthattheDevelopmentOfceshouldconductemployeeevaluationsannually.

Level of Compliance: In Compliance

In conducting the 2007 evaluation of the WVDO, the LegislativeAuditor requested employee evaluations of specic positions within inthe WVDO and found that, although evaluations of some positions werecurrent, other positions had not been evaluated for several years. The2007 report stated:

Given the importance of economicdevelopmentandthelevelofcompensationallocated to the Development Ofce,performancemeasurement and employeeaccountabilityshouldbeofhighpriority.

Although the WVDO and other classied-exempt agencies do notfall under the Division of Personnel, many such agencies tend to followthe Division of Personnels policies, which require annual evaluations.Therefore, the Legislative Auditor recommended that the Divisionconduct evaluations annually.

On April 27, 2010, the Legislative Auditor requested the mostrecent employee evaluations for the same positions obtained for the 2007report. The majority of evaluations received were conducted after thisrequest was sent. Upon inquiry as to whether employee evaluations hadbeen conducted annually following the release of the 2007 report, theWVDO stated the following:

ThepolicyoftheWVDOinpreviousyearswas to conduct employee performanceappraisalsbasedonanemployeeshiringdate. However, due to factors such as

administrativepersonnelleavingforotheremployement, restructuring of the ofceanda change inexecutivedirectors, notevery employee received an employeeperformance appraisal after the releaseoftheSeptember2007PerformanceAuditoftheWVDO.Whileworkplanswereinplace tomonitor employee performance;noteveryemployeereceivedanemployeeperformance appraisal. However, in

Issue 2

In direct response to therequirement that the WVDOconduct annual appraisals or alsta, a directive rom ExecutiveDirector Kelley Goes was sen

out on May 5, 2010, stating thaall sta will be evaluated on acalendar year basis.


16/25


Development Ofce

In the 2007 evaluation, the Legis-lative Auditor recommended thatthe Development Ofce shouldully complete the employeeevaluation orms.

directresponsetotherequirementthattheWVDOconductannualappraisalsforallstaff,adirectivefromExecutiveDirector

KelleyGoeswassentoutonMay5,2010,statingthatallstaffwillbeevaluatedonacalendaryearbasis(January1-December31).

This directive requiring that all staff be evaluated on a yearlybasis brings the WVDO into compliance with the recommendation madein the previous report.

Recommendation 5

TheLegislativeAuditorrecommendsthattheDevelopmentOfceshouldfullycompletetheemployeeevaluationforms.


In conducting the 2007 evaluation, the Legislative Auditorfound that some employee evaluation forms were incomplete, leavingblank pertinent sections that wouldproveusefulinidentifyingobjectivescompletedandareasofconcernduringtheevaluationperiod.Althoughseveral of the evaluations obtained in order to determine compliancewith the recommendation made in the 2007 report were incomplete,the WVDO explained the missing information and indicated plannedcompliance as follows:

The previous director of the SmallBusiness Development Center did notcomplete performance evaluations and,duetothefactthatthecurrentdirectorhasbeen employed for less than sixmonths,it is not appropriate for evaluations tobe submitted at this time; however theSmallBusinesssstaffwillbeevaluatedinJanuary of 2011The previous director

of Community Development departedforanotherpositionandwasnotabletocompletehispartoftheevaluationforonememberofhisstaff.

Evaluations for other departments were fully completed, therebycomplying with the recommendation that the WVDO complete employeeevaluation forms.

Although several o the evalu-ations obtained in order to de-

termine compliance with therecommendation made in the2007 report were incomplete, theWVDO explained the missing in-ormation and indicated plannedcompliance.


17/25



The 2007 evaluation pointed outhat employee perormance wasoten measured by outputs, suchas the number o meetings organized, rather than outcomessuch as new businesses createdWhile the WVDO still primarilyocuses on outputs rather than

outcomes as perormance measures, some outcome measuresare now being utilized.

Recommendation 6

TheLegislativeAuditorrecommendsthatoutcomesof

performancebemeasuredattheemployeelevelandthatsuchmeasuresbeattributabletotheagencysmission.


The 2007 evaluation pointed out that employee performance wasoften measured by outputs, such as the number of meetings organized,rather than outcomes, such as new businesses created. While the WVDOstill primarily focuses on outputs rather than outcomes as performancemeasures, some outcome measures are now being utilized. The WVDOindicates that all projects are

assigned a representative to assist andmonitorthevariousstagesofdevelopment.In the case of theBusiness& IndustrialDevelopment division, either a businessexpansion and retention representativefor existing industry or marketingrepresentativefornewinvestmentintothestateisassignedtotheprospectivebusiness.Additionally,WVDOrepresentativesworkwitheconomicdevelopmententities(state,local,andinternational)tobringaboutthesuccessfulcompletionofeachproject.

The WVDO generates reports to monitor performance thatmeasure such variables as new, current, and retained employment andinvestment resulting from projects handled by WVDO representatives.Additionally, the West Virginia Small Business Development Center(WVSBDC) within the WVDO monitors the impact of counselingservices performed by the WVSBDC staff including economic impactdata such as jobs created and retained, number of businesses created, andthe number of start-up rms remaining in business after one year.

Although measures of employee performance have been improved,there is no indication that the WVDO expands these measures to report

on outcomes at the agency level. As discussed in further detail underRecommendation 1, the WVDO does not measure agency outcomes,instead focusing primarily on inputs and outputs to measure agencyperformance. Since reports are generated that illustrate outcome measuresat the employee level, the WVDO should consider compiling thesegures into overall outcome measures for the agency. For example, jobscreated by projects handled by individual employees could be compiledto illustrate the total number of jobs created by all agency projects.

Although measures o employee perormance have been improved, there is no indicationthat the WVDO expands thesemeasures to report on outcomesat the agency level.


18/25


Development Ofce


19/25



Appendix A: Transmittal Letter


20/25


Development Ofce


21/25



Appendix B: 2009 Trade Show Cost Analysis


22/25


Development Ofce


23/25



Appendix C: Agency Response


24/25


Development Ofce


25/25



Building 1, Room W-314, State Capitol Complex, Charleston, West Virginia 25305

t l h 1 304 347 4890 | l i t t /J i t/PERD/ d | 1 304 347 4939

Documents

The WV Development Office Has Not Improved Performance Measures