Embed Size (px)
Citation preview
Proposal Scoring Plans and Rating Systems
Breakout Session #: G01
Marge Rumbaugh
Date: Wednesday, July 27
Time: 9:45am–11:00am
Overview of Topics Presented
2
• Review Proposal Evaluation Principals • Describe Evaluation Criteria and Weighting • Explain Proposal Scoring Plan Methods • Illustrate How Agencies Weight and Score
Proposals • Provide Lessons Learned from Case
Studies • Summarize Important Points About
Weighting and Scoring Proposals
Proposal Evaluation Principles: The Best Value Continuum
3
• Lowest Price Technically Acceptable – Pass/Fail evaluation – Lowest price wins
• Trade-off process (Best Value) – Evaluates “shades of gray” – Scoring system needs to convey
more complex information.
What Gets Evaluated?
4
• Price or Cost • Quality
– Typically Technical/Management • Past Performance: Over the
Simplified Acquisition Threshold • Small business subcontracting
participation in the subcontracting plan as an evaluation factor.
FAR 15.304
Evaluation Criteria: State Relative Order of Importance
5
• Significantly more important than cost or price,
• Approximately equal to cost or price, or
• Significantly less important than cost or price.
FAR 15.304
Evaluation Criteria Weighting Example
6
Source Selection Decision
Non-Cost Cost
55% 45% Approximately equal
to cost or price
Significantly more important than cost or price
80% 20%
Significantly less important than cost or price
20% 80%
Evaluation Criteria Structure Example
7
Basis of Award
Cost Non–Cost
Technical
Management
Past Performance
Non-Cost Evaluation Criteria Organization
8
1.0 Area: Management
Broad categories that define non-cost
criteria at the highest levels
1.1 Factor: Personnel
1.1.1. Sub-Factor: Personnel
1.1.1. Sub-factor: Personnel Retention
Standard
Minimum level of compliance
for specific factors and sub-factors
Divide into more detailed parts
specific parts of the overarching area
What is a Proposal Scoring Method?
9
• A scoring method or rating system uses a scale of words, colors, numbers or other indicators to denote the degree to which proposals meet the standards for the non-cost evaluation factors.
• Some commonly used scoring methods or rating systems are – Adjectival, -- Color coding, – Numerical, --Combination
Adjectival Scoring Method (1)
10
Adjective Definition
Outstanding
A proposal that satisfies all of the agency’s requirements, with extensive detail indicating a feasible approach & a thorough understanding of the problems. The proposal has numerous significant strengths that are not offset by weaknesses. The proposal has an overall low degree of risk.
Good
A proposal that satisfies all of the agency’s requirements, with adequate detail of a feasible approach & an understanding of the problems. The proposal has some significant strengths or numerous minor strengths that are not offset by weaknesses. The proposal has an overall low to moderate degree of risk.
Adjectival Scoring Method (2)
11
Adjective Definition
Acceptable
A proposal that satisfies all of the agency’s requirements, with minimal detail indicating a feasible approach and a minimal understanding of the problems. The proposal has an overall moderate to high degree of risk.
Marginal
A proposal that satisfies all of the agency’s requirements, with minimal detail indicating a feasible approach and a minimal understanding of the problem. The proposal has an overall high degree of risk.
Adjectival Scoring Method (3)
12
Unacceptable A proposal that contains at least one major error, omission, or deficiency that indicates a lack of understanding of the problems. The approach cannot be expected to meet requirements or involves a very high risk. None of these conditions can be corrected without a major rewrite or proposal revision.
Color Scoring of Strengths & Weaknesses
13
Color Strengths Weaknesses Blue The proposal has
numerous strengths Weaknesses are insignificant & have no apparent impact on the program.
Green Some strengths exist & the strengths clearly offset weaknesses.
A few weaknesses exist; they are correctable with minimal government oversight or direction.
Color Scoring of Strengths & Weaknesses
14
Color Strengths Weaknesses Few strengths exist & the strengths do not offset the weaknesses.
Substantial weaknesses exist & they are correctable with some govt. oversight & direction.
Red There are no beneficial strengths.
Numerous weaknesses exist that are so significant that a proposal rewrite is not feasible within a suitable timeframe.
Color & Adjective Combination Method
15
Color Rating Description
Blue Exceptional Offeror’s proposal has an exceptional understanding of acquisition’s goals
Red Unacceptable Offeror’s proposal has an unacceptable under- Standing of acquisition’s goals.
Yellow Marginal Offeror’s proposal has a fair understanding of acquisition’s goals.
Green Acceptable Offeror’s proposal has an acceptable under- Standing of acquisition’s goals.
Teal Very Good Offeror’s proposal has a very good under- standing of acquisition’s goals.
Numerical Scoring Method
16
Score Description
0 The factor is not addressed, or is addressed in a way that is totally deficient and without merit
1
• There are deficiencies or weaknesses that can be corrected only by significant changes to the proposal. • The factor is addressed so minimally or vaguely that there are widespread information gaps. • The technical evaluation team has serious concerns about the offeror’s ability to perform the required work.
Numerical Scoring Method
17
Score Description 2
• Information is incomplete, unclear, or indicates an inadequate approach to, or understanding of, the factor. • There is a question about the offeror’s ability to perform satisfactorily.
3
• Meets the specifications and requirements. • The offeror could perform & meet the government’s minimum requirements.
Numerical Scoring Method
18
4
• The proposal has some superior features. • Information provided is generally clear, and demonstrates an acceptable ability to accomplish the technical requirements, with the possibility of more than adequate performance.
5 The response to the factor is superior in most features.
Numerical & Adjectival Method
19
8 - 10
0 - .9
1 – 2.9
3 - 5.9
6 -7.9
Excellent
Marginal
Accept- able
Very Good
Range Rating Description Offeror’s proposal has an exceptional understanding of the acquisition’s goals
Offeror’s proposal has very little understanding of the acquisition’ goals
Offeror’s proposal has a very good understanding of acquisition’s goals
Un- acceptable
Offeror’s proposal demonstrates a good understanding of acquisition’s goals Offeror’s proposal demonstrates a poor understanding of acquisition’s goals
Scoring With Evaluation Standards
20
• Standards are the baseline against which the agency evaluates the offeror’s proposals to determine their acceptability and value
• A standard should specify a target performance level for the proposal.
• Standards require evaluators to evaluate proposals against uniform objectives rather than against other proposals.
Scoring With Evaluation Standards
21
Quantitative Standard – Miles per hour – Dollars per pound – Lines of code – Years of experience – Computer processing
speed
Qualitative Standard – Provides an acceptable
technical or management solution/ approach, which meet requirements of the …
– Provides adequate and appropriate personnel skills
– Provides an acceptable level of experience/ knowledge
Scoring With Evaluation Standards: Illustration
22
+ + + Significantly above standards/expectations + + Clearly above standards/expectations + Slightly above standards/expectations
MEETS THE STANDARD - Slightly below standards/expectations - - Clearly below standards/expectations - - - Significantly below standards/expectations
Description
Scoring: Risk Assessment Rating
23
Rating
High
Low
Medium- Low
Medium- High
Offeror’s proposed approach is likely to cause significant schedule disruption, cost increase or performance degradation. It will require significant contractor emphasis & government monitoring to overcome difficulties.
Offeror’s proposed approach is likely to cause minimal schedule disruption, cost increase or performance degradation. It will require a low level of contractor emphasis and govt. Monitoring to overcome difficulties.
Offeror’s proposed approach is likely to cause minimal to moderate schedule disruption, cost increase or performance degradation. It will require a low to medium level of contractor emphasis & govt. monitoring to overcome difficulties.
Offeror’s proposed approach is likely to cause a moderate to significant schedule disruption, cost increase or performance degradation. It’ll require a medium to high level of Contractor emphasis & govt. monitoring to overcome difficulties.
Standards, Scoring, and Risk Rating
24
+ + +Significantly above + +Clearly above +Slightly above MEETS STANDARD -Slightly below standards - -Clearly below - - -Significantly below
Rating
High
Low
Medium
Determine Score based on Strengths &
Weaknesses
Determine Risk Rating
Identify Strengths & Weaknesses
Marginal
Exceptional
Very Good
Acceptable
Unacceptable
Scoring With Weighted Evaluation Factors
25
Non-Cost
Past Perf 50%Technical
Item Item Factor
Factor
Sub Factor
Sub Factor
Factor Factor Factor Factor
Item Item
Sub Factor
Sub Factor Factor
30% Mgmt
50% 50% 75% 25%
75%
25%
50%
50%
25%
25%
25%
25% 50%
50%
60%
40%
Factor Factor Factor
Factor
Factor Factor
Factor
Factor
Scoring Proposals: Individual Evaluations
26
• Evaluators individually assess each proposal in accordance with the RFP’s stated evaluation factors
• Some agencies use standards to help with scoring.
• Evaluators must support adjectival, color, or numerical ratings with narrative statements that explain or justify the given score.
Scoring Proposals: Consensus
27
• After individual evaluations are complete, evaluators meet as a group/team to determine a consensus score.
• The consensus score is an agreement of the strengths, weaknesses, risks, and determine a score for a specific evaluation factor.
Scoring Proposals: Consensus
28
• No simple process exists to help the evaluators reach a consensus rating.
• Evaluators must – Assess the collective impact of
evaluation subfactors on each factor,
– Assess all of the evaluation factors as they relate to each other under the weighting methodology identified in the solicitation.
– Document dissenting opinions
Case Study: Using Different Adjectives
29
• The RFP’s adjectival rating scale: – Exceptional – Very good – Satisfactory – Marginal – Unsatisfactory.
• Actual Rating System used 3 adjectives with a corresponding point system: – Exceptional (3) – Average (2) – Marginal (1)
Trajen, Inc., U.S. Government Accountability Office, B-296334; B-296334.2; B-296334.3 (July 29, 2005).
Case Study: Scoring Methodology Protest
30
• The RFP stated the agency intended to make award on a “best value” basis.
• Technical factors given greater consideration than price.
• If the technical proposals were determined to be essentially equal, price would govern the agency’s source selection decision.
Gap Solutions, Inc. B-310564 January 4, 2008
Case Study: Scoring Methodology Protest
31
• The agency’s proposal scoring scheme was flawed in that it essentially “negated” the technical distinctions among the proposals.
• The effect was to artificially narrow the range of possible total scores making the numeric scores all appear technically equal.
• The award decision was based on low price rather than on technical considerations that should have received more consideration under the RFP’s terms.
Summary
32
• Proposal Scoring is way to summarize evaluation results to assist the Source Selection Authority’s decision making.
• The particular scoring method an agency uses is less important than the consistency with which the evaluators apply the selected method to all competing proposals.
Questions?
33
• Thank you for attending this session!
