DATE: February 29, 2012 RE: Promotional Examination ...media.oregonlive.com/portland_impact/other/Fire... · On October 7, 2011, Lt. Currie met with Scott Fisher at a coffee shop

DATE: February 29, 2012 TO: Fire Chief John Klum FROM: Commissioner Randy Leonard RE: Promotional Examination Protocols As you know, a recent investigation conducted by BHR and Bill Sinnott confirmed allegations that former Division Chief Scott Fisher provided a copy of a Captain’s promotional exam to a prospective future candidate. Although the findings did not identify a specific rule that was violated, I am very troubled by Mr. Fisher’s actions as they serve to undermine the integrity of Portland Fire & Rescue’s promotional system. In the report, Mr. Sinnott and BHR make three recommendations. I am directing Portland Fire & Rescue to implement these recommendations as soon as possible:

1. Increase the security of examination materials by insuring that all draft and final examination documents, both electronic and paper copies, are retained by BHR. BHR shall take steps to insure that no party receives a copy of the examination except for BHR personnel and Subject Matter Experts from external agencies designated to assist with promotional examinations.

2. Discontinue the practice of ad hoc practice examinations. If PF&R wishes to offer

practice promotional examinations, they should be developed in partnership with BHR and offered to all prospective promotional candidates.

3. Communicate security measures with prospective candidates to reduce fear of

manipulation of promotional examinations.

Additionally, I am disturbed that Mr. Fisher’s actions do not violate any PF&R and/or BHR rules or policies. Therefore, I am directing PF&R to work with BHR establish rules prohibiting actions like Mr. Fisher’s, as well as generally prohibiting the sharing of any information obtained by PF&R personnel in the examination process, including prohibiting the sharing of questions asked on written promotional examinations. Thank you for your prompt attention to these directives.

Yvonne L. Deckard, Director1120 SW 5th Ave., Rm. 404

Portland, Oregon 97204-1912(503) 823-3572

Fax (503) 823-4156

Offce of Management and Finance Jack D. Graham, Chief Administrative Offcer

TO: Anna KanwitAssistant Human Resources Director

FROM: James Fairchild, SPHRHuman Resources Business Partner

Bil SinnottSecurity Manager

RE: Report on Investigation Findings

DATE: February 24, 2012

Introduction

On October 11, 2011, Lt. Caleb Currie informed Fire Chief John Klum that retired Division Chief Scott Fisher provided himcopies of promotional tests for Captain and Battalion Chief, including questions and a scored answer key. On October 19,2011, an anonymous letter was sent to the editorial board of the Oregonian with copies to Commissioner Randy Leonardand Bureau of Human Resources (BHR) Director, Yvonne Deckard. This letter implicated Division Chief Fisher andCaptain James (Jamie) Klum in cheating on promotional examinations. Commissioner Leonard assigned former PortlandPolice Commander and current Water Bureau Security Manager, Bil Sinnott, along with Division Chief Glen Eisner andHuman Resources Business Partner (HRBP) Jim Fairchild, to conduct the investigation.

Scope of the Investigation

Initially, the investigation centered on two allegations:

1. Did retired Division Chief Scott Fisher violate city rules or a written security agreement by giving Lt. Caleb Curriematerials for the Captain and Battalion Chief promotional examinations and interviews?

2. Did disclosure of this information to Currie compromise the integrity of future Captain or Battalion Chief promotionalexaminations?

To determine if Lt. Currie's complaint described an isolated incident or was evidence of a more widespread problem withthe testing process, Fire Chief Klum issued a memo to all Portland Fire & Rescue (PF&R) members on December 22,2011. In this memo, he requested any member with information regarding impropriety in the bureau's promotional testingprocess to come forward. Chief Klum provided the contact information for Bill Sinnott, Catherine Riffe, Chief Deputy CityAttorney, and Jim Fairchild, and identified these individuals as persons to whom information should be directed.

Three individuals came forward with concerns. One of the reports was anonymous. This one page anonymous lettercontained personal attacks directed at Mr. Fisher, retired Gresham Chief, Jim Klum (father of Capt. Jamie Klum), Capt.Jamie Klum and Fire Chief John Klum (brother of Jim, uncle of Jamie) and did not further the investigation.

Based on Chief Klum's memorandum, the investigation expanded to include the following two additional issues:

Sam Adams, Mayor

We are an equal opportnity employerPlease notify the City of Portland of the need for ADA accommodations no les than five (5) days prior to any

City-sponsored event by contacting the Bureau of Human Resources at 503-823-3572 or the City's TT at 503-823-868.

Memo to Anna KanwitFebruary 24, 2012Page 2

3. Did Captain Jamie Klum receive any unfair advantage when taking Battalion Chief promotional examination 10-185?

4. Were there any improprieties in the bureau's current or past promotional testing practices?

How the Information Came to PF&R's Attention

A. Lieutenant Caleb Currie's Involvement

On October 7, 2011, Lt. Currie met with Scott Fisher at a coffee shop. Mr. Fisher had retired from his position as DivisionChief of Emergency Operations with PF&R on October 4, 2011. The purpose of the meeting was primarily social;however, during that meeting, retired Chief Fisher gave Lt. Currie an envelope containing the following:

1. A thirt-six page document with the heading "2011 Captains Written Exam Questions." This document containedmultiple choice questions with the proper answers keyed.

2. A two page document with the heading, "Fire Captain Promotional Examination, Panel Interview, February 9,2010." This document contained six interview questions. It was not keyed with answers.

3. A three page document with the heading, "Battalion Chief Promotional Examination, Chiefs' Interview, February 9,2010." This document contained eight interview questions. It was not keyed with answers.

4. A one page document with the heading, "Oral Presentation." This document contained six questions. It was notkeyed with answers.

5. A letter addressed to Scott Fisher dated April 28, 2011, from the Oregon Department of Human Services containingthe original copy of an Emergency Medical Technician renewal certificate #112057 in the name of Scott W. Fisher.

6. A one page color photograph of a dog determined to belong to Chief Fisher.

On the same day he received these documents; Lt. Currie contacted Captain Kristine Artman and asked her for guidanceregarding what he should do with them. Captain Artman did not give Lt. Currie an answer but took the matter underadvisement. She sought advice from Captain Sarah Boone, Senior Fire Inspector David Flood and Senior Fire InspectorAnjanette Jackson. Each of these individuals advised Capt. Artman that the matter must be reported and the testingmaterials turned over to the Fire Chief. Capt. Artman relayed this advice to Lt. Currie on October 10, 2011.

On October 11, 2011, Lt. Currie contacted Fire Chief John Klum and asked for a meeting. The two met later that sameday. Lt. Currie informed Chief Klum of the materials he received from Mr. Fisher and turned these materials over to the

Fire Chief.

Lt. Currie stated that he did not show these materials to anyone other than Capt. Artman and Chief Klum. He furtherstated that he did not make or distribute any copies of these materials.

B. Anonymous Letter to the Oregonian

On October 21,2011, BHR received an anonymous letter dated October 19, 2011 addressed to the Oregonian EditorialBoard with copies to Commissioner Randy Leonard and Yvonne Deckard, BHR Director.

This letter referenced the testing materials given to Lt. Currie by Scott Fisher and stated, without specifics, that, "Therehas been speculation over the years that exam information was shared with certain people at the Bureau..." The letteralso stated that the writer has, "little confidence that the Chief wil do anything with this information...," because, "... theChiefs nephew is currently ranked number four on the Battalion Chiefs list." The writer hoped his letter would "spark aninvestigation that will uncover and bring to light cheating on promotional tests at Portland Fire & Rescue."

C. Background

Memo to Anna KanwitFebruary 24,2012Page 3

Investigators were assigned during the week beginning October 24, 2011. They met on October 27, 2011, to outline theapproach to this investigation. Initial interviews were conducted on November 1, 2011.

D. Interviews

Caleb CurrieScott FisherKristine ArtmanSarah BooneKeri CaffreysDavid Flood

Anjanette JacksonJamie Klum

Joseph WahlTammy WiletKyle WochnickWitness 1

Witness 2

E. Documents Reviewed

Staff LieutenantDivision Chief, PF&R (retired)CaptainCaptainSenior Human Resources Analyst, BHRSenior Fire InspectorSenior Fire InspectorCaptainManager, Employment & Development Section, BHRAdministrative AssistantStaff Captain

ComplainantRespondentWitnessWitnessWitnessWitnessWitnessWitnessWitnessWitnessWitnessWitnessWitness

The investigators reviewed the written test questions, assessment center questions, interview questions, score sheets andThe Bureau of Human Resources Security Agreements for the 201 0 Battalion Chief Promotional Examination (10-185)and the 2011 Fire Captain Promotional Examination (11-003), Battalion Chief Expectations dated June 2008, theanonymous letter sent to the Oregonian, Chief Klum's memo to the bureau dated December 22,2011, an anonymousletter sent to Mr. Sinnott, and the material Mr. Fisher gave to Lt. Currie.

F. Statement of Lt. Caleb Currie

Lt. Currie acknowledged that he approached Mr. Fisher in the months before he retired asking for guidance regarding thepromotional examination process. Lt. Currie considered Mr. Fisher a mentor. As a newly appointed Lieutenant, Curriefrequently asked Mr. Fisher for guidance with tactical situations and management challenges. As time progressed and Lt.Currie developed his own management style, he relied on Mr. Fisher less and less. He did, however, continue to see Mr.Fisher as a mentor and friend.

When he asked Mr. Fisher for assistance in preparing for the Captains promotional examination, Lt. Currie did not ask forexamination questions and did not expect to receive any. He stated that he has never received questions or answers inthe past when he asked for assistance and knows of no one who has. Rather, Lt. Currie expected more genericassistance such as guidance on how to study and how to prepare a resume.

When Mr. Fisher turned over the examination materials, he told Lt. Currie not to show the testing materials to anyone ortell anyone he had them. Lt. Currie felt uncomfortable and knew he should not have the documents. Immediatelyfollowing his meeting with Mr. Fisher, Lt. Currie called Captain Artman and requested a meeting. At that meeting,Lt. Currie showed the documents to Captain Artman. He did not specifically ask Artman for guidance but expected her togive him directions on what to do.

G. Statement of Captain Kristine Artman

Capt. Artman confirmed that Lt. Currie contacted her and asked to meet her off site. When they met, Lt. Currie showedher the documents he received from Mr. Fisher. Capt. Artman confirmed that Lt. Currie told her he received this materialfrom Mr. Fisher earlier the same day.

Capt. Artman said that she knew it was not proper for the documents to be in Lt. Currie's possession. However, as a


probationary Captain, Capt. Artman was conflicted about how to proceed. Even though he had retired, she wasconcerned that Mr. Fisher was still in a position to undermine her career. Given her uncertainty about how to manage thesituation, Capt. Artman sought guidance from three senior supervisors over the next few days. Each of these supervisorsadvised Capt. Artman that the matter must be brought to the attention of the Fire Chief and the testing materials turnedover to him.

Captain Artman phoned Chief Klum on October 10, 2011, and informed him of the situation. She then advised Lt. Curriethat she had spoken with Chief Klum.

Capt. Artman stated that Mr. Fisher helped her prepare for a promotional examination by conducting a mock assessmentcenter including an interview and in/out basket exercise. She stated that she received no actual test questions and knewof no one who had other than Lt. Currie.

H. Statement of Senior Fire Inspector (SF!) Anjanette Jackson

SFI Jackson stated that Capt. Artman approached her seeking advice. Capt. Artman told her that she had beenapproached by an unidentified Lieutenant who showed her promotional test questions received from Mr. Fisher. Capt.Artman was concerned about how best to get the facts out given the senior rank of personnel involved.

SFI Jackson told Capt. Artman that she had to report the matter. SFI Jackson later followed up with Capt. Artman toverify that she had made the report.

SFI Jackson was formerly assigned as PF&R's Recruiting Offcer. She stated, "if someone asks me ... what do youremember from your Lieutenant's test, I would share what I remember, what i had to do, but I think that's vastly differentthan handing over a file of all the tests..."

SFI Jackson believes that certain people have received improper assistance on examinations. She has no specificknowledge of this occurring. Her suspicion is based on her lack of trust in Mr. Fisher and the close relationship he haswith the Klum family.

i. Statement of Capt. Sarah Boone

Capt. Boone stated that Capt. Artman approached her asking for advice on how to respond to the information receivedfrom Lt. Currie. Capt. Boone was "quite concerned about the integrity of the (testing) process," when she heard that Mr.Fisher gave Lt. Currie testing materials.

Capt. Boone advised Capt. Artman that she must inform the Fire Chief.

J. Statement of SFI David Flood

SFI Flood stated that Capt. Artman approached him and informed him that Lt. Currie was in possession of testingmaterials received from Mr. Fisher. She asked SFI Flood for guidance and he replied that she must inform the Fire Chief.

SFI Flood's perspective is that the bureau "monkeys" with the Battalion Chief eligibility list. He believes the problem was"widespread under Fisher" and further stated that he does not believe "Fisher is ethicaL." SFI Flood added, "i am relativelycertain without having anything to back it up, other than my prior knowledge of Scott, that this has been going on to hisfavored crew of younger people."

K. Statement of Retired Division Chief Scott Fisher

Mr. Fisher considered it his' responsibility to mentor any member who wanted to promote. Prior to being appointedTraining Division Chief (TDC), he helped members prepare by conducting mock assessment centers and by providingpast interview questions, practice examinations and sample resume formats. He estimated that he assisted between fortyand fifty people in this manner over the course of his career, including Kristine Artman for whom he administered a mockassessment and practice written test before her first Lieutenants promotional examination.


After he was appointed TDC, Mr. Fisher stopped providing mentoring or assistance because he was, "an integral part ofthe process." Mr. Fisher never gave examination materials or actual test questions to anyone other than Lt. Currie.

Mr. Fisher was asked how he developed the practice materials he gave to those he mentored. In response, he stated thatfollowing the conclusion of promotional tests, assessment centers and interviews, he routinely met with candidates andrecreated all the questions they could remember. . Using these questions, he created practice exams, mock interviews andassessment centers. He often critiqued the resumes of candidates and showed them alternate formats. Mr. Fisher also

. suggested areas on which to concentrate studies including General Orders, Chiefs Memos, and published informationregarding fires in other jurisdictions, performance management and fire ground tactics. He stated that this kind of generalinformation is formally provided in the informational sessions that the TDC and BHR conduct before every promotionalexamination.

Retired Chief Fisher admitted that he gave Lt. Currie the documents described above. "i gave the material to LieutenantCurrie on or about October 3rd." At the time, Mr. Fisher did not think he was doing anything improper because PF&Rnever uses the same promotional test twice. Each test is unique. The questions on each test are individual and varied.Mr. Fisher also believed that no examination material could be given out during a testing period but after the test has beenconducted, "the material isn't confidential any longer." In retrospect, Mr. Fisher believes that releasing the testingmaterials was a mistake. He stated, "... in retrospect, yes, I probably shouldn't have given it to Caleb Currie, but what Ihad didn't come from HR. I wasn't a part of that written process. Anyone in this organization could have asked andgotten pretty much I think what I got."

Mr. Fisher did not believe that he cautioned Lt. Currie not to show the testing materials to anyone or tell anyone that hehad them. He indicated that "I might have said I don't really want to be involved in other people helping them promote,because I really wasn't interested in coaching and guiding other people now that I'm retired..."

Mr. Fisher strongly denied that he gave any improper assistance to any individual or manipulated test results. He pointedout that some of his best friends in the bureau did not do well on promotional exams and certain other people whom hefelt were not qualified for promotion did welL. He believed this was evidence that he did not influence the outcomes ofexaminations because had he done so, the results would have been different. Mr. Fisher specifically denied assistingCaptain Jamie Klum to prepare for promotional exams.

Mr. Fisher played no role in creating the questions for the 2011 Captain promotional written examination. He helpeddevelop the assessment center for the 2011 Captain promotional examination. He did not give any information about theassessment center to Lt. Currie because of his involvement in creating it.

L. Statement of Witness 1

Witness 1 was one of two members who responded to Chief Klum's December 22, 2011 memo. Witness 1 stated that hetook the Captain's promotional examination about two and one-half years ago. In the assessment center associated withthat examination, there was an exercise in which the candidate was given a headset and a laptop computer. Theapplicant listened to a recording of a simulated voicemail in which the Chief forwarded a complaint left by a citizen. Thecandidate was instructed to write a memo stating what action should be taken in response to the complaint. Whenfinished, the candidate was to create a folder and save his or her work. Witness 1 wrote what he believed was, "probablythe best memo of my career." Each of the three evaluators gave him a score of 7 out of a possible 12 for this exercise.None of the assessors for this section made any written comments regarding their scores. Because the assessors for theother sections made comments, Witness 1 concluded that this section of the examination was, "manipulated."

Witness 1 stated that the written test portion of this exam contained 116 questions. The test was open-book. He lookedup the answers for "each of the first 1 00 questions" and was certain that he chose the right response for all but two ofthese questions. The two questions, for which he could not locate the answers, were later eliminated from the final scorefor this section of the test. Witness 1 concluded that he should have achieved a perfect score on the test and the fact thathe did not is evidence that the test was at best inaccurate.

Witness 1 also indicated that he and one other person missed the deadline to submit an application for this test. He


believed that the examination was not properly announced. He had to appeal to be allowed to sit for the exam.

Witness 1 recalled that sometime in 1979, there was a problem in the Training Division with recruits stealing tests andanswer keys. He said that this incident was never investigated although it was reported to those in authority. Shortlythereafter, an individual was caught in the act of stealing a test. In reaction to this incident, there was "a shake up" at theTraining Division and procedures were changed to ensure better security.

When asked directly if he knows about anyone directly involved in cheating on examinations, Witness 1 replied, "...1 haveno specific knowledge of anybody in particular..."

M. Statement of Witness 2

Witness 2 came forward in response to Chief Klum's memo. Witness 2 expressed concerns regarding the integrity ofBattalion Chief promotional examination 10-185 and about Capt. Jamie Klum's placement on the eligible list.

Witness 2 was one of the subject matter experts (SME) who constructed a tactical exercise for the assessment center.Witness 2 said that he put a great deal of time and effort into constructing a realistic tactical scenario involving three roleplayers, a candidate's handbook outlining the procedures to follow, benchmarks for assessing the candidates'performance and scoring ranges with a narrative for the assessors to read and decide where each candidate fell on therange.

Witness 2 attended a meeting prior to the examination. Also in attendance were Chiefs Fisher and Eisner as well as KeriCaffreys from BHR. The purpose of the meeting was to discuss the exercise Witness 2 created. When Witness 2described his plan for three role players, "Chief Fisher became irate." Chief Fisher insisted that there be only one roleplayer and stated that he had previously informed Chief Eisner of that decision. Witness 2 felt that the exercise would notbe as realistic with only one role player. He interpreted Chief Fisher's insistence that there be only one role player as anattempt to narrow the number of witnesses to each candidate's performance. .

Following the administration of the assessment center, Witness 2 encountered one of the assessors. He was curiousabout how his tactical exercise went and interested in knowing how he might improve it. The assessor reportedly toldWitness 2 "the diffcult part was that there was no scoring... There were benchmarks but no score... There was no way toattach a score to your candidate." Witness 2 knew this was not the same exercise he created. This concerned him.

Witness 2 spoke with another of the assessors who told him "there was a candidate that would not have passed aLieutenants' tactical in our organization," who placed well on the list for promotion. That candidate was Capt. Klum.

Witness 2 identified one additional area of concern. Shortly before the administration of the examination, Capt. Klumconducted a Rapid Intervention Team (RIT) dril at Station 24. Witness 2 indicated that the crew found the timing of thisdril interesting because it was out of character for Capt. Klum to conduct RIT drills. RIT was part of the tactical scenarioon the assessment center.

Witness 2 knows Mr. Fisher and Capt. Klum's father are friends and that there exists a friendly relationship between Mr.Fisher and the extended Klum family. Witness 2 became suspicious that Mr. Fisher altered the tactical scenario, alertedCapt. Klum to the RIT component of the test and otherwise influenced the outcome of the examination to assist Capt.Klum.

When asked if he had any direct knowledge that Mr. Fisher or anyone else assisted Capt. Klum, Witness 2 answered, "Ido not." When asked if he had any direct knowledge of cheating on any PF&R tests, Witness 2 answered, "The only thingI know is what has been discussed in the media, but I have no particulars on any cheating of any type."

N. Statement of Staff Captain Kyle Wochnick

Captain Wochnick was assigned to the Training Division in 2004 and remains in that assignment today. During histenure, Capt. Wochnick provided oversight and coordination for SMEs developing several promotional examinations andhas drafted examination questions himself. Capt. Wochnick stated that each promotional examination is distinct from all


others. It is not possible for SMEs to review previous examinations before drafting questions. Neither is it possible forSMEs to use questions from previous tests. This is because no copies of previous tests or draft questions are retained atthe Training Division. After the proposed questions are submitted to BHR, all drafts and working copies are destroyed.Additionally, Capt. Wochnick stated that repeating questions from previous tests does not make sense because thecontent of the study material changes significantly between tests.

When asked about security during test construction, Capt. Wochnick stated that he keeps the door to his offce shut andthe blinds closed when he works on questions. He maintains the data on a password protected computer which is lockedin his offce. Should he use a CD or flash drive to store data, he keeps it locked in a cabinet inside a locked offce. Capt.Wochnick said that he is so concerned about security that, on occasion, he has returned to work after going home toverify that he locked up the draft testing materials. To back up his data, Capt. Wochnick gave discs containing draft testquestions his Administrative Assistant, Tammy Willet. He instructed Ms. Willet to safeguard this data and allow accessonly by designated SMEs. After the draft questions were submitted to BHR, Capt. Wochnick said that he retrieved thedata from Ms. Wilet and destroyed it.

Capt. Wochnick stated that he did not give examination materials to anyone other than the staff of BHR, the SMEs and hisAdministrative Assistant. Capt. Wochnick admitted that he gave the document entitled "2011 Captains Written ExamQuestions," to Scott Fisher. This is the same document later given by Mr. Fisher to Lt. Currie. Capt. Wochnick stated thathe gave this document to Mr. Fisher during the time this examination was under development. Capt. Wochnick is certainof this because he destroyed copies of the draft questions for that examination after they were submitted to BHR. Capt.Wochnick did not believe it was improper to give these questions to Scott Fisher because Mr. Fisher was his supervisorand also an SME involved in the development of promotional examinations.

O. Statement of Administrative Assistant Tammy Willet

Tammy Wilet confirmed that Capt. Wochnick gave her copies of examination questions under development as a way ofbacking up the data. Ms. Wilet stated that the information was locked in a cabinet in her work area or stored on apassword protected PC. Ms. Wilet said that she is the only person who knows the password for her PC and that she hasthe only key to the cabinet. .

Ms. Willet did not allow anyone other than the subject matter experts to access examination questions. Ms. Willet deniedmaking any copies of this data. She said that some draft questions from previous examinations were still on her PC.

P. Statement of Captain Jamie Klum

Captain Klum stated that he has been preparing for the Battalion Chief promotional examination his entire career, "... onething about Battalion Chief is you start early in your career, as far as preparation goes, so it's not something you preparefor in a couple months." To prepare, he completed the Supervisor/Manager training series offered by BHR and studied allthe Human Resources Administrative Rules (HRAR). He was also a member of the bureau's first Cultural AssessmentCommittee and completed the Culturally Competent Manager training series conducted by BHR. He accepted a variety ofadministrative and station-based assignments including Captain of the land-based marine response team where hecreated the land-based response program, logistics chair for the Maritime Fire and Safety Association (MFSA) and StaffCaptain assigned to Battalion Headquarters.

As the date for the promotional examination drew nearer, Capt. Klum "prepared by going over Operational Guidelines,SOPs, Chiefs Orders, everything that was available to me and anybody else to prepare for the job." He also studied thebureau's General Orders, strategic plan and a ten page document entitled, Battalion Chief Expectations, authored byDivision Chief Mark Schmidt in June of 2008. Regarding the latter, Capt. Klum stated, "If you read that whole book, it tellsyou pretty much how to walk and talk like a Battalion Chief... there's a lot of good stuff in there, so if you're trying to readsomething and put it into memory so that you can go into these assessment centers and deliver, you're gonna do as muchas you can and you're gonna start abiding by what's in the book, regardless if I'm a Captain or Battalion Chief."

When asked why he conducted a RIT dril prior to the promotional examination, Capt. Klum responded that the BattalionChief Expectations directs incumbents to understand all policies and procedures relative to RIT and to be prepared todeploy RIT teams when Firefighters are in danger. Capt. Klum took this expectation to heart and conducted a dril to


familiarize himself and the fire company with RIT procedures.

Over the years, Captain Klum, has consulted with Mr. Fisher regarding how to approach certain operational situations.Capt. Klum also obtained a copy of Mr. Fisher's resume in order to learn the format Capt. Klum denied ever receivingany testing materials from Mr. Fisher or anyone else. He stated that he received no assistance in preparing for theBattalion Chief promotional exam.

The Promotional Testing Process for PF&R

Information regarding the promotional testing process was obtained from Joseph Wahl, Manager of the Employment &Development Section (E&D) for BHR and Keri Caffreys, Senior Human Resources Analyst (SHRA) for BHR. Theydescribed the process used for promotional testing in PF&R as follows.

All entry level and promotional examinations for PF&R are developed and conducted by the Bureau of Human Resources.PF&R provides technical assistance through SMEs who create draft test materials and provide logistical support beforeand during the testing process.

The PF&R hiring manager contacts BHR requesting a promotional examination. PF&R promotional examinations aretypically administered every two years for each sworn classification. Promotional opportunities are always internalrecruitments open only to current PF&R members.

Once notified that the bureau intends to conduct an examination, the SHRA meets with the TDC and together, they reviewthe classification specification. They identify the critical knowledge, skils and abilities (KSAs), essential to the job.

Using the KSAs as a foundation, the SHRA and the TDC determine the type of examination they will conduct (e.g. writtentest, assessment center, panel interview). They also establish when the application period opens, when it closes,certifications required for eligibility, the number of years experience required for eligibilty and any other requirements forpromotion. All of this information is used to develop a job announcement. Once approved, the announcement is postedto notify anyone who wishes to apply.

As applications begin to arrive in BHR, the SHRA conducts a first level screening to determine if the applicant possessesthe minimum qualifications described in the announcement. Those individuals who demonstrate that they meet theminimum qualifications are permitted to sit for the examination.

After the application period closes but before the examination is administered, the SHRA and the TDC conduct aninformational session. All applicants for the examination are invited to the informational session. The TDC makes apresentation to the applicants about what they can expect during the test. No actual test questions are revealed but themethods by which the candidates will be evaluated are identified. For example, if there is going to be an in/out basketexercise or a fire scene role play, the applicants receive this information. The applicants also receive study materials. Allthe test questions and exercises are developed from this study materiaL. Anyone who has access to this material hasaccess to all the information that wil be covered in the exam. Additionally, the TDC describes the KSAs the bureau islooking for in the ideal candidate. The SHRA answers questions such as when the eligible list is posted and how thescores are computed. This informational session is recorded for people who cannot make it to the meeting in person.

Creation of Examination Content

Each promotional examination is unique and different from all the others. Questions are developed specifically for eachexam. To create test questions and assessment center activities, the TDC identifies a limited number of SMEs, usuallytwo or three. These SMEs sign a Security Agreement affrming that they wil maintain strict confidentiality with regard tothe testing materials in their possession.

When an assessment center is to be conducted, the SMEs create role play exercises and other hands-on activities.These can include simulated fire scenes where candidates must make decisions and respond to unanticipateddevelopments, business writing exercises, personnel management problems, in/out baskets, responses to mock citizencomplaints, panel interviews and similar activities that assess the candidate's abilty to apply the KSAs identified for the


job. The SMEs submit the assessment center activities they create to the SHRA who ensures that they apply to thedimensions of the classification specification. The SHRA may modify the exercises to reduce the number of variableswhich could result in different candidates being faced with slightly different cues when they complete the exercise. TheSHRA also may modify the suggested scoring criteria if it is overly complex and could lead to confusion among theassessors. It is very important that the assessors can uniformly score the candidates' responses.

The actual assessment center exercises and scoring criteria are not available to anyone in PF&R until the day theexamination is administered.

If the examination contains a written test, a very similar procedure is followed. Using the identified study materials, theSMEs create a large number of multiple choice questions. These questions are submitted to the SHRA who reviews themto be sure they cover the dimensions of the classification specification, that they are based on information in the studymaterials, that they are well worded and not overly ambiguous and that one of the multiple choices is the proper answer.The analyst may reorder questions that appear to be similar to questions used before. The SHRA then selects a subset ofthe questions submitted for inclusion on the test. The questions selected are not revealed to anyone at PF&R before theday the test is administered.

SHRA Caffreys and Mr. Fisher were independently questioned about changes made to the tactical exercise submitted byWitness 2. Both were in agreement that the exercise as designed was too complicated to be reliably conducted. BHRreduced the number of role players from three to one in order to improve the likelihood different candidates face the samecues and respond to the same set of variables. The scoring criteria submitted were too complex for the assessors to usereliably. A simpler set of criteria was developed. Ms. Caffreys disputed the suggestion that there was no way forassessor to score the candidates. A review of the actual score sheets submitted by the assessors confirmed theexistence of numerical scoring criteria. All of the assessors utilized those criteria.

Examination Administration

Promotional examinations are administered by BHR with assistance from PF&R and outside assessors.

A. Battalion Chief Promotional Examination #10-185

The announcement for this examination was posted on November 22,2010. The application period closed on December6, 2010, and the information session for candidates occurred on January 3, 2011.

The examination was divided into two sections, an assessment center and a paneL. interview. The assessment center wasa pass/fail test designed to determine the readiness of candidates for promotion. Only candidates who received a passingscore on the assessment center progressed to the panel interview. All candidates who progressed to the interview arrivedequally ranked. Scores from the interview were used to determine the eligible list.

The assessment center consisted of four exercises, an apartment fire tactical scenario, a warehouse fire tactical scenario,a Rapid Intervention Team (RIT) presentation and a personnel management scenario. Three assessors rated eachcandidate on each of the four exercises. Each candidate received three scores for each exercise and a total of twelvescores for the assessment center. Anyone assessor's score equaled one twelfth of the candidate's total score. Theassessors for this promotional examination were two Battalion Chiefs from Gresham Fire, one Battalion Chief from thePort of Portland Airport Fire Department, three Battalion Chiefs from Tualatin Valley Fire & Rescue (TVF&R), twoBattalion Chiefs from Clackamas Fire & Rescue, one Battalion Chief from Hilsboro Fire & Rescue and one non-sworncitizen of Portland. Assessors received training in how the exercises were constructed and how the scoring criteria wereto be applied. They independently rated each candidate on his/her performance based on scoring criteria provided byBHR.

Each scenario was conducted in a separate room with candidates rotating between rooms. The assessors assigned toeach exercise remained the same throughout the examination process. In each room was a proctor employed by BHR.Proctors received instruction on how to conduct the assessment exercise to which they were assigned, how to apply thescoring criteria and how to respond to any problems or questions that arise during the examination. Each proctordistributed testing and scoring materials separately for each candidate and collected completed score sheets from each


assessor at the close of the exercise. Scoring sheets for each exercise are color coded and constructed in a specificformat to assure that no one is able to tamper with the results. Colors for the scenarios vary from test to test so it is notpredictable which color will be used for which event. After each candidate completed the exercise, proctors collected thescoring materials. The proctor reviewed the scores to verify that all score sheets were complete and that there were noproblems with assessors using the established scoring criteria.

At the completion of the assessment center, all the scoring sheets and testing materials were collected by arepresentative from BHR who verified that all the materials were handed in, that all scoring sheets were complete and thatthere were no problems with assessors using the established scoring criteria. Score sheets were not shown or distributedto any employee of PF&R.

BHR tabulated the scores for each candidate and created a numerical distribution of scores. This distribution did notreveal the name or any identifying information about the candidates who achieved specific scores. It consisted only of theindividual scores arranged from high to low. The numerical distribution was then submitted to PF&R. BHR worked withthe hiring manager to determine the pass/fail point. Candidates with scores above the pass/fail point moved on to panelinterview while those who scored below it did not. It was only after the decision about which candidates would continueand which would not, that the names of the successful and unsuccessful candidates were provided to PF&R. AlthoughCaptain Klum did not achieve his best scores in the tactical portion of the assessment center, his overall scores placedhim in the group that moved on to the panel interview.

All candidates arrived at the panel interview equally ranked. The assessment center and the interview scores were notcumulative. Rather, the assessment center determined which candidates demonstrated readiness for promotion and theinterview scores determined their placement on the eligible list. The panel interview consisted of four questions. Eachcandidate's responses were independently evaluated by three raters. Raters for this interview were Mark Schmidt,Division Chief; Jeff Bancroft, Deputy Chief and Jay Guo, Senior Business Operations Manager. Raters received trainingin how the questions were constructed and how the scoring criteria were to be applied. Raters independently scored eachcandidate on his/her performance based on the criteria provided by BHR.

The interview process was proctored by an employee of BHR. This employee distributed scoring sheets to the raters andcollected them at the end of the interviews. Score sheets were checked to verify that each rater completed one sheet foreach candidate and to ensure the raters applied the scoring criteria consistently.

Interview scores were tabulated by BHR independently of input from PF&R. Once these scores were tabulated, aneligible list was certified by BHR and provided to PF&R.

B. Captain Promotional Examination 11-003

SHRA Caffreys said that the written component of Captains promotional examination 11-003 was administered on May25, 2011. The second phase of this examination was an assessment center which was conducted in June 2011. CaptainKyle Wochnick was the SME who worked with SHRA Caffreys to develop the written test questions for this examination.As is customary, SHRA Caffreys reviewed the questions submitted by Capt. Wochnick and constructed a written testusing a subset of these questions. She confirmed that document 1 given to Lt. Currie was not a copy of the test whichwas administered. She believes that it was a draft of questions prepared by Capt. Wochnick. Mr. Fisher stated that heobtained this document from Capt. Wochnick. These draft questions which Mr. Fisher gave to Lt. Currie were given wellafter the Captain exam had been administered.

Manager Wahl was asked if there is any City which rule prohibits drafts of test questions from being given to individuals asstudy guides after the examination concluded. He answered that he was unaware of such a rule.

Confidentiality of Examination Materials

All persons having access to testing materials before or during an examination are required to sign a Security Agreement.This includes SMEs that assist in the development of test content, assessors who score candidates during assessmentcenters, and raters who score candidates during interviews. By signing this agreement, individuals acknowledge that theyare personally responsible and accountable for all examination materials in their possession. They affrm that they wil not


make, retain, or distribute copies of the materials; that they wil secure the materials when not in use; they wil not discussthe materials or permit them to be viewed by persons not directly involved in the development or administration of theexamination; they will not take the materials home; they will not revise the materials or administer them outside of theexamination for which they were developed and that they wil return all examination materials and notes to BHR at thecompletion of the examination. Mr. Fisher signed a copy of this Security Agreement when he participated in thedevelopment of examinations. BHR personnel who oversee test development and administration are held to a higherstandard of confidentiality.

The Security Agreement does not contain a sunset date, however, according to SHRA Caffreys, the agreement is onlybinding until the end of the testing period. She stated, "Once the test is over, they can talk about it." When asked if it is aviolation for someone to distribute testing materials after they signed a Security Agreement, Ms. Caffreys stated, "if it wasdone during the testing period. After the testing period is over, no."

Manager Wahl was asked if there is a policy addressing the practice of senior PF&R members conducting practiceassessment centers for individual candidates. He stated, "Not that we were aware of.... As long as it's not happeningduring the recruitment or testing process. That would be considered giving somebody probably unequal treatment. Justlike when we run a recruitment here and somebody asks if we can look at their resume and cover letter, once therecruitment opens, no, sorry. If it was something they just approach us on at a different time, when that particularrecruitment wasn't open, we could help them out. But once that recruitment opens, you're not supposed to... assist peoplein testing." Mr. Fisher expressed a similar understanding of the Security Agreement when he stated, "After the test isdone, the written test is completed, the assessment center is over, that material isn't confidential any longer."

Findings and Conclusions

1. Did retired Division Chief, Scott Fisher violate city rules or a written security agreement by giving Lt. Caleb Curriematerials for the Captain and Battalion Chief promotional examinations and interviews?

Conclusion: Mr. Fisher did not violate any city rules or the written security agreement by giving these materials to Lt.Currie.

Mr. Fisher acknowledged giving Lt. Currie the materials. Document 1 is a draft of test questions prepared by Capt.Wochnick for Captain's Examination 11-003. Document 2 appears to be practice questions recreated by Scott Fisher.(There was no Captain's examination given on February 9,2010.) Document 3 appears to be practice questions recreatedby Scott Fisher for the Chiefs interview related to Battalion Chief Examination 10-185. Document 4 appears to be from amock assessment center constructed for a practice test. These documents are not the actual test or interview questionsused in the Captain or Battalion Chief examination or selection process.

There is no city rule which prohibits drafts of test questions from being given to individuals as study guides after theexamination concludes.

The Security Agreement signed by Mr. Fisher contains no sunset provision and would appear to prohibit the retention anddistribution of materials from previous tests. However, both Manager Wahl and SHRA Caffreys, who have authority overthe testing materials and process, stated that confidentiality must be maintained over testing materials before and duringthe examination period. Once the test has been administered and scored, the confidentiality requirement ends. Mr.Fisher gave the documents to Lt. Currie after the examination period had ended.

2. Did disclosure of the information Mr. Fisher provided to Lt. Currie compromise the integrity of future Captain orBattalion Chief promotional examinations?

Conclusion: There is no evidence that future Captain or Battalion Chief examinations have been compromised.

The disclosure was limited. Lt. Currie did not copy the materials Mr. Fisher gave him and turned the materials directly overto Chief Klum. Moreover, BHR retains the actual examinations and maintains their confidentiality.

3. Did Captain Jamie Klum receive any unfair advantage when taking Battalion Chief promotional examination 10-185?


Conclusion: There is no evidence that Captain Klum received any unfair advantage on Battalion Chief promotionalexamination 10-185.

Witness 2 raised a number of concerns that led him to believe that Capt. Klum received an unfair advantage on BattalionChief promotional examination 10-185: (1) changes made to the tactical exercises; (2) comments from an assessorregarding Capt. Klum's performance on one section of the assessment center; (3) his final rank on the promotionaleligibility list, which was seemingly at variance with the assessor's comments: (4) conducting a RIT drill shortly before thetest; and (5) Mr. Fisher's close relationship with Capt. Klum's father and uncle.

Current Chief Training Offcer, Glen Eisner, and Witness 2 were the bureau members who were responsible for draftingthe assessment center. Mr. Fisher was asked to do a dry run of the assessment center, but otherwise his role waslimited. .Regarding construction of the test, SHRA Caffreys and Mr. Fisher were independently questioned about changes made tothe tactical exercise submitted by Witness 2. Both were in agreement that the exercise as designed was too complicatedto be reliably conducted. The number of role players was reduced from three to one in order to improve the likelihooddifferent candidates face the same cues and respond to the same set of variables.

The scoring criteria submitted were too complex for the assessors to use reliably. A simpler set of criteria was developed.Ms. Caffreys disputed the suggestion that there was no way for assessors to score the candidates. The investigatorsreviewed the actual score sheets submitted by the assessors and confirmed the existence of numerical scoring criteria.All of the assessors utilized those criteria.

The raw test data shows that Capt. Klum's assessment center scores placed him within the group of candidates whomoved forward from the assessment center to the panel interview. The tactical exercises were not where Capt. Klumdemonstrated his greatest strength; however, his combined scores for the four assessment center exercises resulted in apassing score. Capt. Klum scored very well on the panel interview portion of the examination.

Regarding the RIT dril, Captain Klum said that he reviewed a document entitled Battalion Chief Expectations to preparefor BC Exam 10-185. This document includes the following instruction:

Understand without question, all policies and procedures related to firefighter safety. This includes, but is not limitedto, callng for a 2"d RIT company, launching RIT, standardized terms for emergency scene management (Abandon,Withdraw, Evacuate, etc), Emergency Traffic procedures (with tones), air monitoring, etc, etc, etc. Most companyofficers are aware of these policies or procedures, but for a Battalion Chief they must all be second nature. It wil bethe Battalion Chief on scene that everyone looks to when the unusual or catastrophic event happens. BEPREPARED.

This document supports Capt. Klum's statement that he conducted a RIT drill in order to prepare himself for thepromotional examination.

There is no evidence that the test scores were manipulated, or could have been manipulated by anyone at PF&R, in sucha manner to favor Captain Klum. Additionally, all of the assessors were from outside agencies. Capt. Klum firmly denieshe received any inappropriate assistance and no witness has come forward with evidence to contradict his assertion.

4. Were there any improprieties in the bureau's current or past promotional testing practices?

Conclusion: The investigation revealed no individual or systematic cheating or improprieties on promotional tests.

Only two members responded to Chief Klum's memo. Thejr allegations were thoroughly investigated and notsubstantiated. In addition, the evidence regarding the testing process established that PF&R could not manipulate scoresto favor or disfavor specific candidates. BHR procedures ensured the integrity of the examination process at each stage.


Recommendations

1. Security

The BHR Employment & Development team instituted additional security measures for the Lieutenant promotionalexamination currently under development. These measures require SMEs to develop draft questions in the BHRoffces using BHR computers. No draft questions for this examination exist on any PF&R computer. No electronic orhard copies of the drafts are permitted to leave BHR. We recommend that this become the standard practice used inall future promotional examinations.

No drafts of questions or assessment center events used for construction of promotional examinations should ever bedistributed to anyone other than SMEs and BHR staff. All copies of such drafts should be turned over to BHR.

2. Practice Sessions

This investigator became aware that several fire offcers have organized improvised practice sessions for candidateswho wished to promote. These practice sessions were conducted under the banner of mentoring and included mockassessment centers, interviews and resume screening. There are problems with this practice. First, there is noassurance that all candidates for a promotional examination have equal access to the practice sessions. We found noevidence that any candidate who requested inclusion was denied but we also found no evid,ence that all candidateshad the option of participating in these practice sessions. Further, we are concerned about the quality of thesepractice sessions. With no oversight and no quality control in place, candidates may have very different experiencesdepending on the skil of the offcer conducting the session. The usefulness of these sessions may vary significantlydepending on the mentor. Therefore, we recommend the following:

The practice of ad hoc practice sessions for promotional examinations should be discontinued. If PF&R desires toprovide practice sessions, a practice assessment center and/ or interview could be designed and conducted by theBHR staff working in conjunction with designated PF&R SMEs. All candidates for the examination should be notifiedof the opportunity. Materials used in these practice sessions should be clearly marked "Practice" and all copiesthereof retained by BHR. No practice sessions should be conducted between the date the application period opensand the date the eligibie list is posted. SMEs who participate in the development of draft questions and/orassessment center events shall be prohibited from conducting practice events for the examination on which theyworked. Candidates who wish to form study groups on their own may do so.

3. Communications

We found that a lack of understanding regarding the way examinations are constructed and administered contributedto rumor and suspicion. We recommend that the SHRA clearly inform the SMEs drafting questions for written teststhat they are contributing to a pool of questions, all of which will not be used and many of which will be reworded.Further, the SHRA should clearly inform SMEs contributing to assessment center events that the events and scoringcriteria submitted may be modified to create an activity that can be uniformly administered to all candidates and hasscoring criteria that are not overly complex and can be consistently applied by all assessors. Additionally, it isrecommended that the SHRA use the informational session to notify candidates of the security safeguards built intothe testing process. If the candidates were aware that no one at PF&R has access to the examination before it isadministered and that pass/fail decisions are made based on an anonymous distribution of scores, it may allay fearsthat the results are subject to manipulation in favor of specific candidates.