Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Title VII Case Study:
Plaintiff v. U.S. City
Defending Interviews Against a Title VII
Adverse Impact Claim
February 16, 2011
Visit BCGi Online
• If you enjoy this webinar,
– Don’t forget to check out our other training opportunities through the BCGi
website.
• BCGi Standard Membership (free)
– Online community
– Monthly webinars on EEO compliance topics
– EEO Insight Journal (e-copy)
• BCGi Platinum (paid) Membership ($199/year)
– Fully interactive online community
– Includes validation/compensation analysis books
– EEO Tools including those needed to conduct AI analyses
– EEO Insight Journal (e-copy and hardcopy)
– Members only webinars and training and much more…
www.BCGinstitute.org
HRCI Credit
• BCGi is an HRCI Preferred Provider
• CE Credits are available for attending
this webinar
• Only those who remain with us for at
least 80% of the webinar will be eligible
to receive the HRCI training completion
form for CE submission
44
About Our Sponsor: BCG
• Assisted hundreds of clients with cases involving Equal Employment
Opportunity (EEO) / Affirmative Action (AA) (both plaintiff and
defense)
• Compensation Analyses / Test Development and Validation
• Published: Adverse Impact and Test Validation, 2nd Ed., as a practical
guide for HR professionals
• Editor & Publisher: EEO Insight an industry e-Journal
• Creator and publisher of a variety of productivity
Software/Web Tools:
– OPAC®(Administrative Skills Testing)
– CritiCall®(9-1-1 Dispatcher Testing)
– AutoAAP™ (Affirmative Action Software and Services)
– C4™ (Contact Center Employee Testing)
– Encounter™ (Video Situational Judgment Test)
– AutoGOJA®(Automated Guidelines Oriented Job Analysis
®)
– COMPare: Compensation Analysis in Excel
Title VII Case Study:
Plaintiff v. U.S. City
Defending Interviews Against a Title VII
Adverse Impact Claim
February 16, 2011
Contact Information
Dan A. Biddle, Ph.D.CEO
(800) 999-0438 x 113
6
AgendaAgenda
7
• Interviews in the Courts
• Characteristics of defensible Interviews
• Adverse Impact review
• Case Review
• Validation Process
• Case Outcome
DisclaimerDisclaimer
8
• Discrimination, in both “allegation” and “court
concluded” forms, is a serious issue.
• BCG works on both sides of discrimination
lawsuits and believes that Title VII currently
provides the means for conducting a balanced
investigation of adverse-impact based
discrimination to be investigated.
• Validation should be a process used for building
fair and effective employment devices, and not as
a means for “just defending tests that are
potentially discriminatory.”
Interviews and the Courts
• The question is still sometimes asked…
– “Are Interviews really tests”?
– Yes, they are really tests
• Any Practice, Procedure, or Test (PPT) that separates two groups (e.g., men/women) based on two possible outcomes (e.g., pass/fail) is classified as a “test” under the Uniform Guidelines
Interview Defensibility:
Some General Characteristics…
UnstructuredUnstructured StructuredStructured
Single Rater Single Rater Multiple RatersMultiple Raters
Generic Generic ““one size fits allone size fits all”” Job SpecificJob Specific
Open Scoring/No Scoring Open Scoring/No Scoring BARSBARS
Least DefensibleLeast Defensible Most DefensibleMost Defensible
Interview Validity:
When Interviews are correlated to job
performance, here’s what we typically see:
Unstructured StructuredUnstructured Structured
rr= .11 = .11 -- .18 .24 .18 .24 -- .34 .34
Low Validity Low Validity High ValidityHigh Validity
Litigation Involving Interviews
• Is there a connection between Interview type and success
in court?
• Williamson et al. (1997). Employment interview on trial:
Linking interview structure with litigation outcomes.
Journal of Applied Psychology, 82 (6), 900-912.
– Study involving 84 disparate treatment and 46 disparate
impact cases where interviews were litigated
– 17 interview characteristics were evaluated (e.g.,
objective, subjective, standardized, etc.).
– Study resulted in clear findings that revealed the three
primary ingredients for successful interview validity
defense
Litigation Involving Interviews: Key
Defensibility Characteristics
• The Three Primary Factors Are…
– Interview objectivity and job relatedness, such as:
o Objective and specified criteria
o Trained interviewers
o Validation evidence
– Standardized administration, including:
o Scoring guidelines
o Minimal rater discretion
o Common questions
o Consistency
– Multiple Interviewers
o Implies a shared decision making process
o Rater reliability
What is Adverse Impact?What is Adverse Impact?
Adverse Impact is defined as a Adverse Impact is defined as a substantially substantially
different ratedifferent rate of selection in hiring, of selection in hiring,
promotion, or other employment decision promotion, or other employment decision
which works to the disadvantage of members which works to the disadvantage of members
of a race, sex, or ethnic groupof a race, sex, or ethnic group
14
Adverse Impact: Selection Rate
Comparison (1991 CRA) & UGESP
Amends Section 703 of the 1964 Civil Rights Act (Title VII)
(k)(1)(A). An unlawful employment practice based on
disparate impact is established under this title only if:
• A(i) a complaining party demonstrates that a respondent uses a particular employment practice that causes a disparate impact on the basis of race, color, religion, sex, or national origin, and the respondent fails to demonstrate that the challenged practice is job-related for the position in question and
consistent with business necessity; OR,
• A(ii) the complaining party makes the demonstration described in subparagraph (C) with respect to an alternate
employment practice, and the respondent refuses to adopt such alternative employment practice.
15
Adverse Impact
Selection Rate Comparison
2 X 2 Table
Used to evaluate hires, promotions and terminations
Statistical Significance:
• 5%
• 0.05
• 1 chance in 20
• 1.96 Standard Deviation
16
Now Let’s Review the Specific
Recent Interview Case…
• Claim from a female minority member holding “Position I” that the city discriminated against her when she applied to be promoted to “Position II”
– The claim of discrimination was based on race using an “Adverse Impact” approach
• A structured oral interview process was used as a primary means of selection
• Plaintiff claimed selection process was not valid (i.e., job related and consistent with business necessity)
• While this case occurred in the public sector context, the process and concepts are identical for private sector employers covered by Title VII
Four-Fifths (80%) Rule of Thumb
• Court indicated that a violation of
the 80% rule of thumb alone is not
sufficient to establish a prima facie
case
• A statistically-significant difference
in selection rates needs to be shown
Position II Selection HistoryPosition II Selection History
Year Status Focal Group Reference Group
2001 Hired x x
2001 Applied x x
2003 Hired x x
2003 Applied x x
2005 Hired x x
2005 Applied x x
2007 Hired x x
2007 Applied x x
• Data from different testing sessions
needs to be combined appropriately
How Should the Data Be Combined?How Should the Data Be Combined?
• Add the years together?
• Conduct adverse impact for each year
and average?
• Use a special statistical analysis
specifically used for combining different
versions of interviews across year?
– Hint! Multiple Event Probability
Test (MEEPT)
SimpsonSimpson’’s Paradox Examples Paradox Example
Simpson's paradox is an apparent paradox in which a correlation
(trend) present in different groups is reversed when the groups are
combined.
Using Appropriate Statistics
• Hiring combined
by simple adding
• Multiple Events
Exact Probability
Test
MEEPT Years
Fisher Exact Mid-P
Probability Value
2001, 2003, 2005, 2007 0.088
2001, 2005, 2007 0.147
Pooled Years
Fisher Exact Mid-P
Probability Value
2001, 2003, 2005, 2007 0.127
2001, 2005, 2007 0.235
The “true” relationship is different for different interviews across
a period of time
PlaintiffPlaintiff’’s Statistical Expertss Statistical Experts
• Mistakenly added a minority group member to the analyses who did not participate in the selection process
• BCG found Plaintiff’s statistical expert’s calculation error
• The Plaintiff’s expert admitted they added the numbers incorrectly when conducting their analyses – When correct data was used, they rested their claim on the fact that minority group members were promoted at a rate less than 80% of whites
– However, the difference in selection rates were not statistically significant
What is Validation?What is Validation?
Definition:
• Traditionally, validation is making sure a
practice, procedure, or test actuallymeasures
what it is designed to measure.
• In a legal realm, a selection procedure is valid
if it can be proven by an employer that it is
“job related and consistent with business
necessity.”
24
Why Validate?Why Validate?
• It is good business
– Validation often results in selection devices that are better at helping to select qualified applicants
– Can reduce the amount of time spent training newly-hired employees
– Sends a message to the community and your employees that you care about doing the “right thing”
– Minimizes potentially negative legal actions
• It is required under certain circumstances
– If there is adverse impact against a protected group of test takers
– If challenged in court or during a government audit
25
Job
Duties
Content Validation Process
Operationally
defined KSAOs
Other KSAOs
Selection
Devices
(e.g., Structured
Oral Interview) Where these
three
overlap is content
valid
Job Analysis: The Foundation for CV Job Analysis: The Foundation for CV
DefensibilityDefensibility
Low: Rely on job
descriptions with
no job analysis
about what is
required on the
job; no links
between duties,
KSAOs, and PPTs
High: Full &
comprehensive
job analysis,
including input
from job experts &
supervisors; links
between duties,
KSAOs required at
entry, and PPTs
How the Job Analysis was ConductedHow the Job Analysis was Conducted
Reviewed Historical
Documents
Feedback from
Incumbent Employees
Individual Telephone
Interviews
Job Analysis
Job Duties
KSAPCs
Surveyed
Job
Expertsusing
AutoGOJA
Analyzed Data
Important/Critical Job
Duties
Important/Critical
KSAOs required upon
entry
Job Analysis was Conducted RemotelyJob Analysis was Conducted Remotely
• A BCG consultant conducted telephone
interviews of Position II incumbents (SMEs),
their supervisors, and a Human Resource
staff member
• A search of the literature was made
regarding the positions
• Previous job postings and job descriptions
were examined
What Ratings Does BCG Collect During What Ratings Does BCG Collect During
a Contenta Content--related Job Analysis?related Job Analysis?
• For Job Duties we collect ratings for:
– Frequency* (Guidelines, 14.C[2])
– Importance* (Guidelines , 14.C[2])
– Differentiating* (Guidelines , 14.C[2])
– Relative difficulty
– Fundamental (Americans with Disabilities Act)
– Assignable (Americans with Disabilities Act)
* = required under federal Guidelines
• For knowledge, skills, abilities, and other
characteristics (KSAOs) we collect ratings for:
– Frequency* (Guidelines, 14.C[2])
– Importance* (Guidelines, 14.C[2])
– Performance Differentiating
– Minimum v. Desirable Qualifications
– Level needed upon entry
– Level needed for success (knowledge only)
What Ratings are Collected during a What Ratings are Collected during a
ContentContent--related Job Analysis?related Job Analysis?
* = required under federal Guidelines
Targeting Key KSAOs for Test Building
• Only certain types of KSAOs can be tested when using content-related validity
o The PPT measures and is a representative sample of a knowledge, skill, or ability and is
o Used in and is a necessary prerequisite to performance of critical or important work behaviors (i.e., shown through a link to job duties)
o Should closely approximate an observable work behavior, or its product should closely approximate an observable work product
• Skill, abilities, & other characteristics being measured should be “operationally defined” in terms of observable work behaviors
Identify Job Duties
Rate Job Duties
Identify KSAOs
Rate KSAOs
Link Important Job Duties and KSAOs
Link Important Job Duties & KSAOs
to Selection Device
Develop Selection Device
Content Validity Flow Chart
Structured Interviews
• Interview questions asked of job candidates were based on an examination of the job– Job Analysis and validation study confirmed the job-relatedness
of the questions
– Job Analysis foundation rested on a review of city documents, telephone interviews, and ratings collected from a panel of Job Experts
• Interview panels were trained by City’s Office Human Resource Agent
• Consensus discussions by the raters were held after each interview was completed
• Candidates were rank ordered based on scores– Effectively used to band – not strict rank order- candidates
• Different oral interview questions were asked of candidates during each year of the selection process
Validity Findings
% of Acceptable Questions
# Testing Standard 2001 2003 2005 2007
1. Is the question CLEAR and UNDERSTANDABLE? 100% 100% 100% 100%
2. Is the question written at an APPROPRIATE DIFFICULTY LEVEL? 100% 100% 100% 100%
3.Is this question FAIR to all groups of candidates (based on their
race, age, and/or gender)?90.90% 100% 100% 88.89%
4.Is the question job related (does it represent a situation that has
occurred or is very likely to occur on the job)? 100% 100% 87.5% 88.89%
5.
Will applicants be able to provide a sufficient response withoutpossessing job-specific knowledge that they will receive during training once they are hired as a Position II?
100% 100% 100% 100%
6. What KSA is measured by this question?
All of the interview questions were linked by the Job Experts to at least one important or critical
KSA.
DonDon’’t Questions Have to Linked to t Questions Have to Linked to
Important or Critical Job Duties?Important or Critical Job Duties?
• Section 14C(2) of the federal Uniform Guidelines indicates that the work behaviors being measured “should be critical work behavior(s) and/or important work behavior(s) constituting most of the job.”
• An interview question to important or critical work behavior (Job Duty) linkage can be inferred in the current study since each interview question was linked to KSAs and we had Job Experts indicate the strength of the linkages between KSAs and Job Duties
Plaintiff Also ClaimedPlaintiff Also Claimed……• Claim: A separate validation study was needed
before scores from interviews could be used for “rank ordering” of candidates
• Response: The process of ranking and/or score banding is permissible under the federal Uniform Guidelines and does not need to be validated using a “separate validation study.”
• Rather, the process can be validated using the job analysis and validation evidence that exists from the current study. The Uniform Guidelines allow for using a selection procedure for ranking job candidates if the user can specify evidence showing that a higher score on the selection procedure is likely to result in better job performance (see Section 14C9).
Even More ClaimsEven More Claims……
• Plaintiff moved to exclude BCG as the City’s Expert based on the fact that the job analysis and validation was conducted after the litigation commenced
• Plaintiff claimed that since the consultant who was facilitating the job analysis did not visit onsite, it was invalid
• Plaintiff claimed interview questions were written to avoid knowledge and skills and to “cater to the favorites who had been pre-selected”
• The job analysis and validation study was conducted after the claim of discrimination was made
• Even though courts and auditing agencies are likely to more strictly scrutinize validation studies that are conducted after adverse impact is determined, this court ruled that it was acceptable to conduct the study after a claim of discrimination was made
EveEve--ofof--Trial StudyTrial Study
The Court RuledThe Court Ruled……
• Court ruled that to successfully challenge the validation study, Plaintiff would need to submit expert testimony or other objective evidence that tended to refute the findings of the study
– Plaintiff did not provide any expert testimony to rebut the City’s validation study
– Plaintiff did not challenge qualifications of Dr. Biddle or the reliability of his methodology or analyses
– Based on these failures, the court denied the Plaintiff’s motion to exclude this study
Adverse Impact During a Specific Test or Adverse Impact During a Specific Test or
During the Entire Process?During the Entire Process?
• Court indicated Plaintiff must identify a
specific test, requirement or practice that has
an adverse impact on race
– Smith v. City of Jackson, Miss., 544 U.S. 228, 229
(2005)
– Plaintiff failed to identify a specific employment
practice that had adverse impact
• For this reason, the court focused only on the
Interviews given for the position, rather than
the promotional process as a whole
Take Take AwaysAways……
• Selection procedures should be job related
(i.e., valid) and fair to all qualified applicants
• Evidence of validity should be available for
review if challenged
• Written records of testing should be
maintained
• The appropriate statistical tests should be used
• Having EEO experts in your corner can really
help to defend your testing practices
Helpful Validation ResourcesHelpful Validation Resources
• Federal Uniform Guidelines on Employee Selection Procedures
– www.uniformguidelines.com
• U. S. Department of Labor’s Testing and Assessment: An Employer's Guide to Good Practices
– www.onetcenter.org/dl_files/empTestAsse.pdf
• Adverse Impact and Test Validation: A practitioner’s guide to valid and defensible employment testing – Author: Dan A. Biddle, Ph.D.
43
44