CONSULTANT DEBRIEFS - Texas A&M University · 2017 PEPS Conference CONSULTANT DEBRIEFS 2017 PEPS Conference Gail Eileen Morea, P.E.│ Norma Glasscock, P.E. November 3, 2017

2017 PEPS Conference

CONSULTANT DEBRIEFS2017 PEPS Conference

Gail Eileen Morea, P.E.│ Norma Glasscock, P.E.

November 3, 2017


Table of Contents

Objectives

Consultant Debriefs and What They Should Cover

Types of Debriefs: Individual and Group

The Rules of Engagement for Consultant Debriefs

Tips for Successful SOQ

Tips for Successful Interview

Questions and Discussion

11

22

33

44

55

66

77


Objectives

Define consultant

debriefs

Discuss when

debriefs should be scheduled

Discuss debrief

outcomes


Objectives

Definition of a consultant debrief

– The opportunity to share information about a specific procurement in a planned and controlled, formal environment.

– Essential to the continuous learning process

When should a debrief be scheduled?

– Disqualified firms are notified at the screening stage via an e-mail identifying the missing or incomplete items that led to their elimination from further evaluation

– Processes without interviews, once selections are made

– Processes with interviews, after the shortlist is announced

– May be held up to four months after contract execution but the goal is the earlier the better


Objectives (continued)

Outcome of a debrief

– Factual information about the performance of the provider

– Demonstrates that a thorough, fair evaluation based on sound decisions took place in accordance with the selection methodology developed prior to advertisement.

– Open and honest feedback to assist the provider in improving their future submissions in response to new procurements.

– Identifies strengths and how to learn from mistakes going forward.


What should consultant debriefs

cover?


Consultant Debriefs - What Should They Cover?

Reference Solicitation Number, Discipline and the Selection Process followed

Overall high score, low score, the provider’s overall score and their ranking within the solicitation pool itself

The average score for each question in the SOQ and/or Interview

The firm’s individual score by question

Stress the importance of the weighting factor for each question

Identify questions that warranted more information



Was the question answered thoroughly or were the responses too broad brush?

Lessons learned by handling similar projects

Implemented innovative techniques

How was the role of the Project Manager described in the response to a question? Was it one of managing the project, budget, and schedule along with demonstrating their problem solving abilities or was it more of a design role?



o Reference that interview time should be used wisely.

o The interview gives the CST the opportunity to consider the team dynamics and interaction.

o Avoid disclosing what was looked for on a point by point basis when addressing responses.


Consultant Debriefs

How is past performance evaluated?

Adjusted Evaluation Score Average (ESA) based on past performance on TxDOT contracts (5-year window) using a 10-point system

Full credit is no longer given to Project Managers without past performance scores in the TxDOT database

The CCIS Software application provides the ESA and it is then adjusted to a 5-point score based on the 5-year average of all evaluations in the database.

Minimizes potential bias towards firms with TxDOT experience but does not give the PMs lacking TxDOT references an unfair advantage by giving them a perfect score.


Types of Debriefs

Individual (face to face)

Group

Conference call Webex

Debrief can be held as:

Preferred methods: Individual or Group DebriefDuration: 30 minutes to 1 hour


Individual Debriefs


Consultant Debriefs - Individual Debrief Preparation

Review the team’s SOQ and CST members score sheets

Request input from the CST if you do not have their notes

Review their screening checklist and identify potential issues that did not result in a disqualification but could provide relevant information

Review the top team or teams’ information and take notes to serve as a comparison.

Comparison is generic with no specific details about another team’sresponses given

Information should be detailed enough to be meaningful and helpful Focus on highs, lows, and patterns General expectations of each question Areas for improvement Trends exhibited by the CST that impacted selection


Consultant Debriefs - Individual Debrief Meeting

Provider should have a copy of their SOQ available at the meeting

Delivered by either the Procurement Engineer or the Service Center Manager

For in-person individual debriefs, only notes taken to prepare for the debrief are allowed in the meeting. No actual selection documents or files are to be brought to the meeting.

Remember that every CST is different

Maintain professional conduct during the debrief

It is not a time to debate the score!

Purpose is to help improve future responses


Consultant Debriefs - Individual Debrief - Example

Yellow Rose Consulting was not selected for a contract and requested a debrief. They ranked 36 out of 72 firms.

To prepare for the debrief the Procurement Engineer or Service Center Manager read and evaluated the top 2 or 3 ranked submittals for comparison.

Findings:

All 3 were well organized and had clear responses

All had headings that followed the main points of the questions

Contained thoughtful use of bullets and dialogue that provided clarity to the response

Provided depth to responses that related to the added value scoring

De-briefer scoring correlated with outcome



De-briefer then read and evaluated the response of the requesting consultant

Found blocks of dialogue without breaks between paragraphs –difficult to follow direction of response

Each question started with a large paragraph of marketing information

Did introduce members of the team through the response, explaining the team expertise related to the question



De-briefer will provide information on average scores per question and the scores of the requesting consultant

Question Q1 Q2 Q3 Q4 Q5

Percentage Wt. 10% 17% 13% 20% 25%

Average score 3.05 3.09 3.26 3.13 3.43

Yellow Rose Consulting 3.33 3.00 3.67 3.00 3.67

De-briefer will:

– Address any items in the screening that may have been noted

– Provide examples of what constitutes “meets” and “exceeds”

– Provide time to address questions from the consultant



Trends that lead to success for the top 3 firms

Immediately started answering the question

Did not have introductory marketing paragraphs (Our firm has 30 years of experience….), though there may be a sentence or two provided in information on expertise related to the topic of the question

All responses above average

Goal: Help the consultant prepare for afuture with successful SOQ responses.


Group Debriefs


Consultant Debriefs - Group Debrief

– Triggered by the number of SOQ respondents

– The presenter is the Service Center Manager

– The Group Debrief focuses on the responses of the top scoring firms

– Procurement Engineer and/or Service Center Manager should prepare their own notes. If the Consultant Selection Team is not participating in the debrief, request their input

– Information that may be covered by a group debrief

Focus on the highs, lows, and patternsGeneral expectations of the questionsAreas for improvementsTrends that impacted the selection



Additional Information to be shared in a Group Debrief:

Number of SOQs received

Number of disqualified firms (if any), and general information about why these firms were disqualified

Range of total scores

Rounded average score by question

Summary of each question, the applicable weighting and examples of what constitutes “meets and exceeds” requirements

The end result……High performing teams lead to more competition!



Can I obtain our team’s scores by question and overall total

score?

Yes, simply e-mail the presenter of the Group Debrief

and request the information

Does a group debrief mean that I am unable to request an

individual debrief?

No, an individual debrief may be requested; however, it will

be scheduled after thegroup debrief

The goal of a group debrief is to provide as much information to all respondents regarding the procurement expectations in a timely manner.

For procurements where there are 50, 60 or even 90 respondents it is not feasible to schedule an individual debrief in a timely manner due to the focus on negotiations for the current procurement. A group debrief enables the timely sharing of information


Rules of Engagement for Debriefs


Rules of Engagement for Consultant Debriefs

The provider may state that their competitor used the same sub who responded to the question with the same response. Though the same sub may be used by both providers this does not mean the same response is in actuality used without modifications.

If a provider asks how a competitor did, the de-briefer should respond that another firm’s information to include SOQs, proposals, interviews, scores or rankings is confidential and is not shared with others.


Rules of Engagement for Consultant Debriefs

• Come prepared to the meeting with your SOQ • Be courteous and professional• Be respectful of others speaking• Be respectful of the time


Helpful Tips


Tips for a Successful SOQ

Read the RFQ thoroughly!

Prior to the SOQ submittal, take advantage of the question period

Watch for the addenda

Schedule the SOQ development plan to allow time to QA/QC your SOQ before submittal

Verify:

•Your administrative qualification status, and those of the sub-providers for the Federal process

•Precertification status of each task leader on a standard work category•If the PTC form cell turns red, investigate•Person’s name matches the name shown in CCIS•Firm names are the actual legal firm names shown in PS-CAMS


Tips for a Successful SOQ (continued)

Address the NLCs completely

•Credentials•Beginning dates, month and year, as well as ending dates, month and year•Direct response for the exact items requested in the minimum requirements

Answer the questions being asked

Be clear, concise, specific, and technical

Don’t waste space marketing

Consider innovative techniques, challenges, and lessons learned


Tips for a Successful Interview and RFP/Interview

Read the ICG or RFP/ICG (Federal process) thoroughly

Focus on

The proposal format and requirements (Federal process)

The interview (and proposal, if Federal) topics as well as the evaluation criteria and weighting

Technical expertise Only! – marketing material will not be evaluated

Anticipate follow-up questions from the CST. These questions clarify and expand on the topics

Expect tough competition

Show team’s leadership between the PM and the task leads


Questions and Discussion


Contact Information

Gail Eileen Morea, P.E.TxDOT | PEPS Houston Service Center Director(713) [email protected]

Norma Glasscock, P.E.TxDOT| PEPS Fort Worth Service Center Manager(817 )[email protected]

Documents

CONSULTANT DEBRIEFS - Texas A&M University · 2017 PEPS Conference CONSULTANT DEBRIEFS 2017 PEPS Conference Gail Eileen Morea, P.E.│ Norma Glasscock, P.E. November 3, 2017