View
5
Download
0
Category
Preview:
Citation preview
2017 PEPS Conference
PRIME PROVIDER EVALUATION2017 PEPS Conference
Thuy Luong, P.E.
November 3, 2017
2017 PEPS Conference
Table of Contents
2
Purpose of Prime Provider Evaluations
Converting ESA score to Past Performance Score
ESA Score for Providers with No Past Performance Scores
Upcoming Changes to Prime Provider Evaluation
Frequently Asked Questions
11
33
44
55
66
Scoring System22
2017 PEPS Conference
Purpose of Prime Provider Evaluations
3
• Have to –required by rule (43 TAC §9.41 (d))
• Want to – used as a multi-purpose tool
Why do provider evaluations?
• Management tool - use it to communicate our level of satisfaction
• Selection tool - past scores are used to compute the ESA, which is used during selection
How are they used?
2017 PEPS Conference
Evaluations - All About Communication
4
At the beginning to identify what is expected
At the beginning to identify what is expected
At logical points to identify and
correct problems
At logical points to identify and
correct problems
At the end - the final evaluationAt the end - the final evaluation
TxDOT communicates on two levels:
TxDOT communicates:
1 - Is the job getting done? 2 - How well is it being done?
2017 PEPS Conference
Evaluations - A Management Tool
The Prime Provider Evaluation Form:
– Provides performance feedback
– Improves contract management
– Improves contract product
It contains a list of criteria and outcome standards
– Evaluates the consultant PM’s skills and abilities as a project manager
– Does not evaluate consultant PM’s skills and abilities as a professional engineer, surveyor, or architect
5
2017 PEPS Conference
Evaluations - A Selection Tool
TxDOT PM enters the evaluation into CCIS
Evaluation Score Average (ESA) is computed in CCIS
The ESA includes a component for:
– the consultant project manager
– the firm
The ESA is used during the selection process at either
– the SOQ evaluation stage, or
– the proposal/interview evaluation stage
Typically, past performance is weighted at 5-10% of the SOQ or interview score or proposal/interview scores
6
2017 PEPS Conference
Scoring System
7
Project manager starts with 100 points
Firm starts with 50 points
For satisfactory performance on a criterion, the base score is given
For less than satisfactory performance on a criterion, points are deducted (e.g. -3, -6)
2017 PEPS Conference
Scoring System (continued)
8
The PM/firm evaluations are divided into three groups:
The first group consists of the evaluations entered into the CCIS at any time from/including the current day to five years prior to the current day.The second group consists
of the evaluations entered into the CCIS at any time from/including five years prior to the current day to
ten years prior to the current day.
The third group consists of the evaluations entered into the CCIS at any time from/including
ten years prior to the current day and earlier.
2017 PEPS Conference
Scoring System (continued)
9
The evaluation scores for Group 3 are ignored entirely. TxDOT is not interested in evaluations 10 years old and
older.
The evaluation scores for Group 2 are averaged. This number is then be averaged with 100.00
The evaluation scores for Group 1 are averaged. (Note: This number will not exceed 100.00)
2017 PEPS Conference
Scoring System (continued)
10
The PM score is calculated as follows. •If there is an average evaluation score available
for Group 1, that number is divided by 12.50 to yield a PM score equal to or less than 8.00. Thus, an average score of 97.00 would yield a PM score of 7.76.
•If there is no average evaluation score available for Group 1, the effective average evaluation score for Group 2 is used. As above, this number is divided by 12.50 to yield the PM score.
2017 PEPS Conference
Scoring System (continued)
11
The firm score is calculated as follows.•If there is an average evaluation score available
for Group 1, that number is divided by 25.00 to yield a firm score equal to or less than 2.00. Thus, an average score of 49.29 would yield a firm score of 1.97.
•If there is no average evaluation score available for Group 1, the effective average evaluation score for Group 2 is used. As above, this number is divided by 25.00.
2017 PEPS Conference
ESA Procedure
12
For a given PM and prime
provider firm, PEPS obtains
the ESA in CCIS
The ESA for each provider is
entered into the scoring
spreadsheet during the selection process
Documentation -An evaluation
report, listing all of the
evaluations used to compute the
ESA is maintained in selection file
2017 PEPS Conference
CCIS Software Application
13
2017 PEPS Conference
PDF lists evaluations used to compute Firm and PM scores
PEPS files the PDF to Selection File
14
2017 PEPS Conference
Converting ESA score to Past Performance Score
ESA Score Past Performance Score
= 10 5
9.52 < ESA < 10 4
9.04 < ESA <9.52 3
8.56 < ESA <9.04 2
8.56 < ESA 1
15
2017 PEPS Conference
Example of Scoring Spreadsheet
16
Past performance is included as a criterion at the SOQ stage, Interview, or Proposal/Interview stage
2017 PEPS Conference
Example of Scoring Spreadsheet (continued)
17
Example of Past performance used at the Proposal/Interview stage:
2017 PEPS Conference
ESA Score for Providers with No Past Performance Scores
For Project Managers (PM) and Firms with no previous TxDOT experience or no evaluations on files, TxDOT uses an average score. This average is calculated using data from the past 5 years. Summary of the data points used:
18
Number of Data Points Used
Score ESA
PM 5 YR Average 752 98.09 7.84
Firm 5 YR Average 770 49.39 1.97
Total (perfect score 150) 147.48 9.81
2017 PEPS Conference
How is the 5-year Average Score Used?
19
Evaluations available
ESA Evaluations available
ESA Evaluations available
ESA Evaluations available
ESA
PM 8* 8* X 7.84 X 7.84
Firm 2* X 1.97 2* X 1.97
10* 9.97 9.84 9.81
P.P. Score 5 4 4 4
* Assuming perfect score for Past TxDOT Experience
2017 PEPS Conference
Upcoming Changes to Prime Provider Evaluation
20
•TxDOT PM completes evaluation in PS-CAMS•TxDOT PM’s supervisor reviews & approves
evaluation •Prime provider PM reviews, comments, and dates
evaluation using PS-CAMS Consultant Portal
New system will be in PS-CAMS
New system will be ready for use in Spring 2018
Scores for each criteria will be from 1 to 5
2017 PEPS Conference
Upcoming Changes to Prime Provider Evaluation (continued)
21
PM Evaluation Criteria
1.a Quality of Deliverables: Accuracy
1.b Quality of Deliverables: Completeness
2 Timeliness of submittals
3.a Contract administration - Budget
3.b Contract administration - Schedule
3.c Contract administration - Invoices
4 Responsiveness and availability of the PM
5 Resolution of issues
6 Communication and coordination
7 Management of subproviders
8 Adequate use and prompt payment of HUB/DBE firms
9 PM performance
Firm Evaluation Criteria
1 Firm Responsiveness
2 Firm Invoicing
3 Firm Resource Management
2017 PEPS Conference
Frequently Asked Questions
What should I do if the TxDOT PM does not evaluate me?
Prime provider evaluation is required:
o Per TAC (43 TAC §9.41 (d))
o Per TxDOT Chief Engineer’s Memo dated April 20, 2017: “Each TxDOT project manager shall complete an end of project evaluation on each consultant contract that they were managing currently. Each TxDOT project manager shall also complete an interim evaluation anytime that a consultant is not performing to expected standards for the Department.”
o PEPS Division assists in this effort by providing reminder email notices to District and Division project managers when contracts are due for an annual evaluations and when contracts are due for end of project evaluation.
22
2017 PEPS Conference
Frequently Asked Questions (continued)
23
Can the TxDOT PM share the evaluation results with the provider PM? That way it
could be a lessons learned or opportunity to do better.
Yes. Once the TxDOT PM has completed the provider
evaluation, he/she must share results with the
provider PM and give the provider PM an opportunity to review and comment on
the results.
2017 PEPS Conference
Frequently Asked Questions (continued)
24
If I don’t agree with the TxDOT PM on the results,
can I not sign the evaluation?
If you don’t agree with the results, you should
discuss with the TxDOT PM and if the TxDOT PM
doesn’t change the scores, you don’t have to
sign the evaluation.
2017 PEPS Conference
Frequently Asked Questions (continued)
25
If I don’t sign the evaluation, will the
scores be included in the TxDOT database?
Yes. The evaluation will be input in the database and the
scores will be included in the ESA
calculation.
2017 PEPS Conference
Frequently Asked Questions (continued)
26
As a provider PM, if I disagree with the TxDOT
PM’s evaluation, how do I protest/dispute the
scores?
First, you should discuss with the TxDOT PM to get the issue resolved. If it
doesn’t get resolved, you may request a meeting
with the TxDOT PM’s supervisor to discuss.
2017 PEPS Conference
Frequently Asked Questions (continued)
27
Why doesn’t TxDOT use the extra points (i.e. +3, +6) when calculating the
ESA?
Extra points are not included because the purpose of the ESA is to identify less-than-
satisfactory performance.
2017 PEPS Conference
Frequently Asked Questions (continued)
28
When TxDOT makes a lot of changes to a project, what must consultants do to get above base
scores on meeting deadlines and staying
within budget?
The consultant should put forth extra effort
to submit quality deliverables and still
meet project deadlines.
2017 PEPS Conference
Frequently Asked Questions (continued)
29
Shouldn’t TxDOT use a small percentage of the
prime provider evaluation score in the selection process since firms and provider PMs
change?
Past performance is weighted at 5-10% during the selection process.
The PM has his/her own evaluation scores and the firm has its own scores. When the PM goes to a
different firm, the PM’s evaluation score goes with him/her.
2017 PEPS Conference 30
Questions and Discussion
2017 PEPS Conference 31
Thuy LuongTxDOT| PEPS Center of Excellence(713) 802-5720Thuy.Luong@txdot.gov
Thuy LuongTxDOT| PEPS Center of Excellence(713) 802-5720Thuy.Luong@txdot.gov
Recommended