HR Avatar Assessment Solution Technical Manual · HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 1 HR Avatar Assessment Solution Technical Manual August 31, 2016

HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 1

HR Avatar Assessment Solution

Technical Manual

August 31, 2016

www.hravatar.com 2810 Thaxton Lane Oakton, VA 22124

703-349-3070


Table of Contents

Overview ........................................................................ 8

Testing Standards .......................................................... 8

Solution Summary .......................................................... 9

Cognitive Work Simulations ........................................ 10

Previous Research ........................................................................................................................................................ 10

Content Development ................................................................................................................................................. 12

Attention to Detail ............................................................................................................................................. 12

Analytical Thinking ............................................................................................................................................ 12

Reliability ....................................................................................................................................................................... 13

Validity ........................................................................................................................................................................... 16

Fairness .......................................................................................................................................................................... 17

Attitudes, Interests and Motivations Assessment ........ 19



Needs Structure .................................................................................................................................................. 21

Innovative and Creative .................................................................................................................................... 21

Enjoys Problem-Solving ................................................................................................................................... 22

Competitive ......................................................................................................................................................... 22

Seeks Perfection ................................................................................................................................................. 22

Develops Relationships ..................................................................................................................................... 23

Expressive and Outgoing ................................................................................................................................. 23

Corporate Citizenship ....................................................................................................................................... 23

Exhibits a Positive Work Attitude .................................................................................................................. 23

Adaptable ............................................................................................................................................................. 23

Reliability ....................................................................................................................................................................... 25

Validity ........................................................................................................................................................................... 25


Fairness .......................................................................................................................................................................... 26

Behavioral History Survey ........................................... 29



Performance ........................................................................................................................................................ 30

Tenure .................................................................................................................................................................. 30

Unproductive Behavior ..................................................................................................................................... 30

Reliability ....................................................................................................................................................................... 30

Fairness .......................................................................................................................................................................... 31

Knowledge and Skills Tests ......................................... 33

Development Overview .............................................................................................................................................. 33

Sales Situation Analysis................................................ 34

Test Development ....................................................................................................................................................... 34

Reliability ....................................................................................................................................................................... 35

Fairness .......................................................................................................................................................................... 36

Typing Test .................................................................. 36


Reliability ....................................................................................................................................................................... 36

Fairness .......................................................................................................................................................................... 37

Essay Test .................................................................... 37


Descriptives ................................................................................................................................................................... 39

Fairness .......................................................................................................................................................................... 39

Solution Scoring ........................................................... 40

Construct Validity Evidence ...................................................................................................................................... 41

Technical Requirements .............................................. 50

Future Research ........................................................... 50

Reliability ....................................................................................................................................................................... 50

Validity ........................................................................................................................................................................... 51

Fairness .......................................................................................................................................................................... 51


Norms ............................................................................................................................................................................ 51

References .................................................................... 52

Appendix A: Summary of the HR Avatar Solutions ..... 57

Appendix B: Historical Validity Evidence for the Cognitive Scales ............................................................................ 72

Appendix C: Historical Validity Evidence for AIMS ... 73

Appendix D: Scoring Rubric for Essays ...................... 78

Appendix E: Directions for Rating Essay Tests ........ 799

Appendix F: Automated Essay Scoring System .......... 82

Appendix G: Validation Studies ................................... 85

Appendix H: Emotional Intelligence Test .................. 88

Appendix I: Work Competency Test ........................... 94


Tables Table 1. Demographic Characteristics of Sample ...................................................................................................... 15

Table 2. Participant Location ......................................................................................................................................... 15

Table 3. Descriptive Statistics and Reliability Evidence for the Cognitive Workplace Simulation Scales........ 16

Table 4. Evaluation of Cognitive Score Differences by Gender ............................................................................. 18

Table 5. Evaluation of Cognitive Score Differences by Age Group ...................................................................... 19

Table 6. Descriptive Statistics and Reliability Evidence for the AIMS Scales ....................................................... 25

Table 7. Evaluation of AIMs Score Differences by Gender .................................................................................... 26

Table 8. Evaluation of AIMs Score Differences by Ethnicity ................................................................................. 27

Table 9. Evaluation of AIMs Score Differences by Age Group ............................................................................. 27

Table 10. Evaluation of AIMs Score Differences by Race Groups: Asian and White ........................................ 28

Table 11. Evaluation of AIMs Score Differences by Race Groups: Black or African American and White .. 28

Table 12. Descriptive Statistics and Reliability Evidence for the Professional Behavioral History Survey ..... 31

Table 13. Descriptive Statistics and Reliability Evidence for the Entry-Level Behavioral History Survey ...... 31

Table 14. Evaluation of Biographical History Survey Score Differences by Gender .......................................... 32

Table 15. Evaluation of Biographical History Survey Score Differences by Ethnicity ....................................... 33

Table 16. Evaluation of Biographical History Survey Score Differences by Age Group ................................... 33

Table 17. Evaluation of Biographical History Survey Score Differences by Race Groups: Asian and White 33

Table 18. Descriptive Statistics and Reliability Evidence for the Sales Situation Analysis Assessment ........... 36

Table 19. Evaluation of Sales Situation Analysis Score Differences ....................................................................... 36

Table 20. Descriptive Statistics and Reliability Estimates for the Typing Test ..................................................... 37

Table 21. Evaluation of Essay Score Differences by Subgroup .............................................................................. 37

Table 22. Descriptive Statistics for Essay Scores ....................................................................................................... 39

Table 23. Evaluation of Essay Score Differences by Subgroup .............................................................................. 40

Table 24. Correlations between the AIMs Scales ....................................................................................................... 42

Table 25. Correlations between the Professional Behavioral History Scales......................................................... 43

Table 26. Correlations between the Entry-Level Behavioral History Scales ......................................................... 43

Table 27. Correlations between HR Avatar AIMs and Behavioral History Scales ............................................... 44

Table 28. Correlations between the Two Cognitive Workplace Simulation Scales: Attention to Detail and

Analytical Thinking .......................................................................................................................................................... 45

Table 29. Correlations between the AIMs and the Cognitive Workplace Simulation Scales ............................. 46

Table 30. Correlations between the Behavioral History Survey and the Cognitive Workplace Simulation

Scales .................................................................................................................................................................................. 48

Table 31. Compiled Validity Evidence for the Original Content of the Cognitive Work Simulation Scales .. 72


Table 32. Study 1 Results: Concurrent Validation Study for Insurance Consultants N=122-136 .................... 73

Table 33. Study 2 Results: Concurrent Validation Study for Inside Sales N=105 ............................................... 74

Table 34. Study 3 Results: Concurrent Validation Study for an Internet Services Order Processing N=84 ... 74

Table 35. Study 4 Results: Concurrent Validation Study for an Internet and Cable Sales and Service Position

N=72-93 ............................................................................................................................................................................ 74

Table 36. Study 5 Results: Concurrent Validation Study Auto Rental Sales Role N=80-92 .............................. 74

Table 37. Study 6: Concurrent Validation Study for Paramedics N=85 ................................................................ 75

Table 38. Summary of Relationships between Performance Measures and Original AIMS Scales ................... 76


Figures Figure 1. Summary of HR Avatar Solution ................................................................................................................. 10

Figure 2. Cognitive Work Simulation ........................................................................................................................... 13

Figure 3. Hypothesized Relationships between the AIMs Scales and the Big Five ............................................. 24


Overview

The HR Avatar Employment Assessment series was designed as a flexible assessment instrument for

helping corporations obtain quality hires by measuring cognitive abilities, biographical data, personality

characteristics and job knowledge related to performance and tenure in the workplace. To cater to differing

individual client needs, HR Avatar offers both individual assessments as well as complete assessment

solutions. By combining assessments and measuring multiple competencies, the assessment solutions allow

for a broader evaluation of overall candidate fit with a particular job.

This technical manual contains a summary of testing standards for the development of psychological

assessments in the workplace and an overall summary of the development of the assessment solution. There

are sections for each assessment within the solution including previous research, content development, and

reliability evidence, validity evidence, and evidence for fairness. The technical manual concludes with a

summary of HR Avatar’s research agenda.

Testing Standards

The Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, National Council on Measurement in Education, 2014), the Principles for the Validation and Use of Personnel Selection Procedures, (Society for Industrial Organizational Psychology, 2003), the Uniform Guidelines on Employee Selection Procedures (Equal Employment Opportunity Commission, C. S. C. U. S. D. L. U., & Equal Employment Opportunity Commission, 1978), and Testing and Assessment: An Employer’s Guide to Good Practices (Saad, Carter, Rothenberg, & Israelson, 2000) are all documents that “govern” the development and use of tests in employment settings. The developers of the HR Avatar Employment Assessment series have made, and continue to make, efforts to adhere to recommendations for test development put forth by these documents. The evidence produced in this technical manual is a summary of all of the theoretical and empirical research done on the tool to date. We anticipate frequent updates as we complete additional research. The following list summarizes the key test development standards from the above listed documents. The documents include additional standards governing test usage and we encourage test users to review and adhere to these standards.

Define the psychological construct measured by the test

Document the intended use of the test

Document how test scores should be interpreted

Demonstrate validity for the specific use of the assessment and interpretation of scores including:

o Previous research on the construct

o Previous empirical research that demonstrates the extent to which the validity evidence may

be generalizable (e.g. meta-analytic studies)

o Research supporting the link between the content of the assessment and the tasks,

knowledge, skills, and abilities required for the job (i.e., content-related validity evidence)


o Research supporting a specific response process takes place if it is inferred in the use or

interpretation of the test

o Evidence that the internal structure of the assessment conforms to theoretical expectations

o Relationships between scores and any outcomes the assessment claims to be able to predict,

such as job performance (i.e. criterion-related validity evidence)

o Evidence demonstrating that the test is statistically related to other measures in ways in

which are consistent with theoretical expectations (i.e., convergent and divergent validity

evidence)

Document sufficient details related to validity research in order to allow the reader an opportunity to

independently evaluate the quality of research

Document reliability estimates and the Standard Error of Measurement (SEM) for each score,

subscore, and combination of scores

Use a type of reliability estimate (e.g., internal consistency, test-retest) that is appropriate for the test

Document sufficient details related to reliability research in order to allow the reader an opportunity

to independently evaluate the quality of research

Design test in such a way that it allows all test takers an equal opportunity to demonstrate their

standing on the construct regardless of subgroup membership (e.g., age, disability status, ethnicity,

gender, age, or race)

Provide empirical evidence demonstrating that the construct is measured the same way across

subgroups

Solution Summary

The HR Avatar Employment Assessment series has been designed for use in identifying or screening out

those job applicants with the lowest potential for success in a given role by assessing job-relevant cognitive

abilities, personality characteristics, behavioral background, and knowledge and skills. There are currently

203 HR Avatar solutions (see Appendix A) in the HR Avatar Employment Assessment series. Each solution

has been designed for specific job roles and contains several components: one of 15 different Cognitive

Work Simulations, one of two different Attitudes, Interests, and Motivations assessment forms (AIMs), one

of two different Behavioral History Surveys, and job-specific Knowledge and Skills assessments. The 15

Cognitive Work Simulations represent 15 distinct job families and are discussed in more detail below. The

AIMs assessment and the Behavioral History Survey are both available in two forms. One form is designed

for professional positions and the other is designed for entry-level positions.


Figure 1. Summary of HR Avatar Solution

The assessments are delivered via a computer or other mobile device via the internet. Applicants are sent an

email with a link to take the assessment. Once the applicant clicks on the link in the email, they are brought

to a page in their web browser. The assessment begins with an animated “host”. The host guides the

applicant through the process and provides instructions prior to the start of each assessment. The animated

host is used to provide the applicant with a more engaging and positive assessment experience. Most

solutions take about 40 minutes to complete.

Prior to implementing the solution for selecting employees, HR Avatar recommends completing a job

analysis to ensure that the competencies measured by the HR Avatar assessments are important for success

in your organization. Additionally, HR Avatar recommends that the organization conduct a pilot study to

examine the relationships between the assessments and relevant job performance criteria and to evaluate the

potential for adverse impact. This pilot study should be done prior to using the assessment for screening out

employees.

Cognitive Work Simulations

Previous Research

Those individuals with higher cognitive abilities are more equipped to solve complex problems and learn

new skills. Cognitive ability is necessary for successful performance on a broad range of tasks including

written documentation, oral communication, identifying and managing details, solving problems, reading

•Measures 1-2 specific cognitive abilities

•1 of 15 job family-specific versions

Cognitive Work Simulation

•10 personlity scales

•2 versions (professional and entry-level)

Attitudes, Interests, Motivations

•2 biographical data scales

•2 versions (professional and entry-level)

Behavioral History Survey

•1 or more of 55 job-specific knowledges tests

•Sales Situation Analysis

•Typing Test

•Essay Test

Knowledge and Skills tests


and responding to email messages, quickly learning and applying new information, quantitative

computations and analyses, and making well-reasoned decisions

The published research evidence consistently demonstrates a strong, predictive relationship between

cognitive ability, or general mental ability, and job performance and training success. In fact, several meta-

analyses have been completed, summarizing and combing the results of many research studies examining

the validity of cognitive ability (Hunter & Hunter, 1984; Schmidt & Hunter, 1998) and the results generalize

across geographical regions (Bertua, Anderson, & Salgado, 2005; Salgado & Anderson, 2003; Salgado,

Anderson, Moscoso, Bertua, & De Fruyt, 2003a; Salgado et al., 2003b). The operational validity estimates

from these meta-analytic studies tend to be in the .5-.6 range. Though cognitive ability predicts performance

at all job levels, the relationship is moderated by complexity such that the relationship is stronger for more

complex jobs (Bertua et al., 2005; Salgado et al., 2003b).

Currently, the most commonly accepted taxonomy for cognitive abilities is the CHC model (Cattell, 1941;

Horn, 1985; Carroll, 1993). The emergence of this accepted hierarchical taxonomy came from the merging

of two independent lines of research examining the taxonomy and structure of cognitive abilities (McGrew,

2008). The taxonomy posits general cognitive ability at Stratum I, broad abilities at Stratum II, and more

specific abilities at Stratum III. For the prediction of job performance, we are most interested in measuring

abilities at the second stratum as these are deemed to be broad enough to apply to multiple positions, but

specific enough to yield incremental prediction in job performance. Furthermore, recent research has

demonstrated that using specific abilities in selection decisions can reduce adverse impact without

decreasing the validity of the selection process (Kehoe, 2002; Wee, Newman, & Joseph, 2014).

The broad abilities listed in Stratum II consist of: fluid reasoning, comprehension-knowledge, short-term

memory, visual processing, auditory processing, long-term storage and retrieval, cognitive processing speed,

decision and reaction speed, reading and writing, quantitative knowledge, general (domain-specific)

knowledge, tactile abilities, kinesthetic abilities, olfactory abilities, psychomotor abilities, and psychomotor

speed (McGrew, 2008). Based on our understanding of these abilities, we believe that fluid reasoning, short-

term memory, reading and writing, quantitative knowledge, and general (domain-specific) knowledge would

be the most generalizable across jobs. However, the general (domain-specific) knowledge was determined to

be better measured by the knowledge and skills assessments. HR Avatar designed two scales for each job

family: Attention to Detail and Analytical Thinking. The development of these scales is discussed further on

the content development section, but, to summarize, Attention to Detail taps short-term memory, whereas

Analytical Thinking is more closely related to fluid reasoning and quantitative knowledge. Both scales

require that applicants read information, therefore, to some extent, the reading component of reading and

writing is also measured by both scales.


Current meta-analytic research has examined the relationship of these specific abilities with overall job

performance and training performance. In a European sample, Salgado et al. (2003b) found that perceptual

ability and memory (similar to the competency measured by the Attention to Detail scale) had corrected

mean validity coefficients of .52 and .56 with job performance and .25 and .34 with training performance,

respectively. Additioanlly, they found that verbal ability and numerical ability (similar to the competency

measured by the Analytical Thinking scale) had corrected mean validity coefficients of .35 and .52 with job

performance and .44 and .48 with training perofrmance, respectively. In a UK sample, Bertua et al. (2005)

found that perceptual abilities had a corrected mean validity coefficient of .50 with job performance and .50

with training performance. Verbal ability and numerical ability had corrected mean validity coefficients

of .39 and .42 with job performance and .49 and .54 with training perofrmance, respectively.

Content Development

The Cognitive Work Simulations measure two cognitive competencies: Attention to Detail and Analytical

Thinking. Attention to Detail most represents short-term memory from the CHC model and Analytical

Thinking is a combination of fluid reasoning and quantitative reasoning.

Attention to Detail

The competency is defined by the ability to process information, recall information accurately, and identify

appropriate resources for locating specific pieces of information. This competency is important for data

entry, identifying errors in data, processing orders, working with tables, understanding policies and

procedures, and working with numerical data such as the type found in financial reports.

Analytical Thinking

The competency is defined by the ability to understand language and the numerical relationships. This

competency is characterized by the ability to process complex information, synthesize data, identify and

solve problems, and make well-reasoned decisions.

The Cognitive Work Simulation requires that candidates complete various job-related tasks in a virtual

environment designed to replicate the workplace. There are 15 versions of the Cognitive Work Simulation

and each is designed to represent scenarios common to a specific job family (listed in Table 3). Candidates

are evaluated on how they respond to avatars representing customers and colleagues and how well they

solve problems in various work-related scenarios. The realistic experience is designed to be more engaging

for applicants and to provide a better measure of ability. During a simulation, the candidate might be

required to read email, listen to voicemail and perform basic keyboard and screen navigation tasks to solve

typical business problems.


Figure 2. Cognitive Work Simulation

The items used in the Cognitive Work Simulation were based on an existing library of ability items with a

history of predicting job performance in various workplace settings. Both the original items and the new

content were developed by examining the results of multiple job analyses. There are between 5 and 19 items

in each competency measured by the simulation. Responses to assessment items are scored dichotomously.

In some items, there is more than one correct response. In these situations, an applicant is awarded a point

for each correct response selected. It takes approximately 18-20 minutes to complete a simulation. The

estimates were calculated using a sample of 264 individuals.

Reliability

To evaluate the properties of the cognitive assessments as well as other assessments in the solution, HR

Avatar conducted a research study. Participants from the study were recruited via Mechanical Turk (MTurk).

MTurk is a platform operated by Amazon.com and allows the user to pay workers for completing for

individual tasks called Human Intelligence Tasks (HITs). MTurk is becoming increasingly popular for use in

research in the social sciences including research related to pre-employment testing. Many researchers have

found that MTurk provided adequate samples for the purposes of developing assessments (Buhrmester,

Kwang, & Gosling; Johnson & Borden, 2012; Minton, Gurel-Atay, Kahle, & Ring, 2013; Miller, Gentile,

Wilson, & Campbell, 2013).


HR Avatar recruited workers on MTurk to complete one of 15 versions of the solution– one for each

version of the Cognitive Work Simulation. The Cognitive Work Simulation was combined with an

appropriate AIMs assessment form, and an appropriate Behavioral History Survey Form. The Knowledge

and Skills assessments were not included in this study. Participants were allowed to take each version once.

Instructions to participants warned them that if they responded to the questions randomly, their HIT would

not be approved, and therefore they would not get paid. Workers were promised an assessment report upon

completion of the assessments. Both of these approaches should have led to more genuine responses.

Additionally, to ensure responses to the assessment were legitimate, the data used in the analyses were

restricted to those individuals who, on average, took 2.5 seconds or more to respond to items (N=675).

These individuals completed 1682 solutions. On average, participants took 2.49 solutions (SD=2.70). A

summary of the demographic information for this sample can be found in Table 1 along with 2013 estimates

(the most recent available) of the demographic breakdown of the U.S. population as reported by the U.S.

Census Bureau (2015). As compared to the U.S. population, the sample in this research contains a higher

proportion of individuals who are male, non-Hispanic and White. The average age of the sample is 33.85

years (SD=10.88) which is also slightly younger than the median age (37.5) of the U.S. population (U.S.

Census Bureau, 2015). Given the differences between the sample and the overall population, the results in

this study are to be regarded as preliminary an interpreted with caution. Table 2 provides a summary of the

geographic areas represented by the sample.

Table 3 contains the descriptive statistics for each scale and the alpha estimates of internal consistency.

Please note, based on these findings, further analyses were done at the item and distractor level. These

analyses led the developers to make substantial edits to these items and we anticipate that edits will

substantially improve the reliability of the tools. Further research evaluating the effectiveness of these

changes is forthcoming.


Table 1. Demographic Characteristics of Sample

Gender N %* U.S. Only 2013 est. from U.S. Census**

Female 287 43.03 43.76 51.4

Male 380 56.97 56.24 48.6

Unknown 8

Ethnicity N %* U.S. Only 2013 est. from

U.S. Census**

Hispanic 46 7.42 6.52 17.1

Not Hispanic 574 92.58 93.48 82.9

Unknown 55

Race N %* U.S. Only 2013 est. from U.S. Census*

American Indian or Alaskan Native 3 .45 .52 .8

Asian 89 13.46 6.55 5.1

Black or African American 25 3.78 4.31 12.6

Native Hawaiian or Other Pacific Islander 5 .76 .69 .2

White 518 78.37 84.14 73.7

Two or More Races 22 3.33 3.79 3.0

Unknown 14

*Percentages calculated by summing those for whom demographic data were available.

**U.S. Census Bureau, 2013 American Community Survey

Table 2. Participant Location

Country N %

United States 589 87.26

India 54 8.00

UK 10 1.48

Other 22 3.76


Table 3. Descriptive Statistics and Reliability Evidence for the Cognitive Workplace Simulation Scales

Cognitive Simulation

Analytical Thinking

Attention to Detail

# of Items

N M SD Alpha # of Items

N M SD Alpha

Basic Entry-Level 19 171 16.68 2.08 0.70 5 171 4.20 1.05 0.53

Face-to-Face Customer Service

6 116 3.80 1.88 0.74 8 116 6.78 1.34 0.53

Administration 5 120 2.62 1.12 0.26 11 119 7.34 2.25 0.67

Business Sales 6 98 3.37 1.53 0.53 8 98 5.96 1.96 0.68

Business/Finance 7 122 3.04 1.36 0.27 6 121 3.63 1.33 0.43

Entry-Level Administration

N/A N/A N/A N/A N/A 10 100 7.38 2.32 0.74

Entry-Level Business & Finance

7 139 2.38 1.84 0.68 6 137 3.83 1.43 0.59

Entry-Level Office 26 142 20.46 4.75 0.86 13 142 9.11 2.71 0.75

First-Line Supervisor 9 102 6.42 1.59 0.47 8 102 5.84 1.85 0.68

General Office Workplace

7 100 4.04 1.63 0.42 6 99 4.67 1.40 0.68

Information Technology

7 98 4.45 1.94 0.65 7 98 5.46 1.30 0.52

Manager 10 98 5.41 1.80 0.38 5 97 3.57 1.16 0.44

Remote Customer Service

6 98 3.45 1.52 0.50 10 98 6.66 2.28 0.71

Technician 8 114 4.77 2.94 0.83 9 114 19.22 3.26 0.73

Retail Sales 10 60 4.97 2.48 0.68 7 60 7.32 1.86 0.66

Validity

As mentioned above, the simulations were developed based on an existing library of content. The original

developer has provided the results of historical validation studies done with the original content and these

are provided in Appendix B.

Validation evidence is available in two studies that demonstrate the validity of the assessments. The first is a

study with 130 managers and the second is a study with 64 managers. In the first study, the assessment

demonstrated that it predicts job performance (r=.25; p<.01), using a simple “High” or “Marginal”

categorization of job performance, and a second study demonstrated the assessment predicted performance

on a 100 point administrative performance appraisal, using a Spearman correlation (r=.25; p<.05). We

expect validity evidence to be more robust in future studies with better criterion measures. A brief summary

of the results is presented in Appendix G.


Fairness

In order to evaluate how different demographic subgroups perform on the assessments, independent

samples t-tests were performed to compare mean scores of the subgroups. The results are provided in Table

4 and Table 5. Included in the tables are the descriptive statistics for each subgroup, the t-statistic (used for

evaluating whether or not the mean differences are statistically significant), and Cohen’s d. Cohen’s d is an

effect size and an indication of how large the mean differences are. For reference, according to Cohen

(1992), effect sizes of .2 are small, .5 are medium, and .8 are large. Although data were collected on ethnic

group, the sample sizes for the ‘Hispanic or Latino’ subgroup were too small for analyses (n<15). Data were

also collected for racial group. However, the sample sizes for all but the “White” subgroup were too small

for analysis (n<30). Future research will examine subgroup differences as more data become available.

For gender, there were statistically significant differences in favor of males on three scales (Entry-Level

Office: Attention to Detail, First-Line Supervisor: Analytical Thinking, and Information Technology:

Analytical Thinking). However, there does not appear to be a strong trend in males scoring higher than

females and it is possible that edits made to the items will diminish differences on the scales where

differences have been detected. For age, there were statistically significant differences on three scales

(Business Sales: Analytical Thinking, First-Line Supervisor: Attention to Detail, and Manager: Analytical

Thinking), however these were all in favor of the protected group. Score differences will be further

evaluated as more data become available on the new version of the assessments and subgroup differences

will be evaluated for ethnic and race subgroups as well.


Table 4. Evaluation of Cognitive Score Differences by Gender

Cognitive Simulation Scale Female Male

n M SD n M SD t d

Basic Entry-Level Analytical Thinking 72 16.57 2.54 97 16.74 1.67 0.52 .08

Attention to Detail 72 4.07 1.18 97 4.32 0.94 1.53 .24


Analytical Thinking 50 3.76 2.06 66 3.83 1.75 0.20 .04


Administration Analytical Thinking 41 2.34 1.15 79 2.76 1.09 1.96 .38


Business Sales Analytical Thinking 41 3.34 1.44 56 3.34 1.58 0.00 .00

Attention to Detail 40 6.10 2.05 53 5.83 1.91 -0.65 -.14

Business/Finance Analytical Thinking 52 2.88 1.28 69 3.12 1.38 0.98 .18




Entry-Level Business/Finance



Entry-Level Office Analytical Thinking 58 19.78 4.67 83 21.02 4.73 1.54 .26

Attention to Detail 58 8.52 3.04 83 9.57 2.38 2.30* .39

First-Line Supervisor Analytical Thinking 42 6.02 1.59 59 6.69 1.57 2.10* .42

Attention to Detail 42 5.97 1.88 59 5.80 1.79 0.46 -.09





Analytical Thinking 42 3.98 1.81 54 4.85 1.98 2.22* .46


Manager Analytical Thinking 36 5.22 1.87 62 5.52 1.76 0.80 .17





Technician Analytical Thinking 26 5.35 2.58 31 4.81 2.46 -0.81 -.21


Retail Sales Analytical Thinking 44 4.41 3.31 69 5.03 2.69 1.09 .21


Note. *p<.05


Table 5. Evaluation of Cognitive Score Differences by Age Group

Cognitive Simulation Scale 40 or Older Less than 40

n M SD n M SD t d

Basic Entry-Level Analytical Thinking 44 16.64 1.73 127 16.70 2.19 0.16 .03





Administration Analytical Thinking 23 2.70 0.76 97 2.60 1.20 -0.38 -.09


Business Sales Analytical Thinking 17 4.41 1.06 81 3.15 1.53 -3.23* -.86


Business/Finance Analytical Thinking 16 2.94 1.24 106 3.06 1.38 0.33 .09





Analytical Thinking 20 2.90 1.80 119 2.29 1.83 -1.38 -.33


Entry-Level Office Analytical Thinking 27 20.52 4.31 115 20.44 4.87 -0.08 -.02


First-Line Supervisor Analytical Thinking 22 6.88 1.08 80 6.29 1.69 -1.55 -.37

Attention to Detail 22 6.65 1.30 80 5.61 1.92 -2.39* -.58







Manager Analytical Thinking 16 6.38 1.36 82 5.22 1.82 -2.42* -.66





Technician Analytical Thinking 13 4.85 2.34 47 5.00 2.55 0.19 .06


Retail Sales Analytical Thinking 26 5.58 2.30 88 4.53 3.07 -1.61 -.36


Note. *p<.05

Attitudes, Interests and Motivations Assessment

Previous Research

The Big Five framework for understanding personality is the most widely accepted taxonomy of personality

traits (Goldberg, 1992; 1993; McCrae & Costa, 1997; McCrae & John, 1992). The five factors, or traits, are

Agreeableness, Conscientiousness, Emotional Stability, Extraversion, and Openness. Agreeableness is

characterized as warmth, ability to get along with others, amd friendliness. Individuals who are


conscientiousness are characterized as being reliable, responsible, dependable, thorough, dutiful,

achievement-oriented and competent. Individuals considered Emotionally Stable are more resilient, stable,

and centered. Extraverted individuals are characterized as being sociable, outgoing, and high in energy.

Openness describes indiciduals who are “creativc, flexible, curious, and unconventional” (Judge & Ilies,

2002)

A substantial amount of research, including meta-analytic research, has demonstrated the relationship

between personality and job performance (Barrick & Mount, 1991; Bartram, 2005; Hogan & Holland, 2003;

Hurtz & Donovan, 2000; Mount & Barrick, 1995; Salgado., 1997; Tett, Jackson, & Rothstein, 1991). In a

meta-analysis of 25 years of personality and performance research drawn from 117 studies of managers,

professionals, sales people, skilled, and semi-skilled workers; researchers concluded that extroversion

predicted success in management and sales; openness to experience predicted training ability; and,

conscientiousness correlated with success in sales, management, professional, skilled and semi-skilled

positions (Barrick & Mount, 1991). The findings were similar in the European community (Salgado, 1997).

Personality traits are stable and slow to change (McCrae, Jang, Livesley, Riemann, & Angleitner, 2001). The

stability of personality traits and their relationship to job performance makes them extremely important to

measure before making a hiring decision.

Theory on personality in the workplace has advanced and researchers have demonstrated that validity

coefficients for personality assessments are stronger for those job performance criteria that can be

theoretically linked to a given personality trait. For example, Borman and Motowidlo (1993, 1997) found

support for their hypotheses that personality would have a stronger relationship with contextual rather than

task performance. The researchers defined contextual performance as those areas of performance that are

important to an organization but not explicity part of the job duties of an individual (e.g., following

organizational rules, giving additional effort when needed, providing support to other organizational

memebers, etc.). Hogan and Holland (2003) found that the Hogan Personality Inventory, a personality

assessment based on the Big Five, was substantially related to multiple measures of job performance across

a wide variety of positions. The relationships were stronger for personality traits that were theoretically

linked to the job performance measure. Bartram (2005) had similar findings, supporting the notion that

theoretically linked personality constructs and job performance domains had stronger relationships than

personality traits with overall performance. The trait-activation model proposed by Tett and Burnett (2003)

provides a theoretical model allowing for more detailed predictions regarding when a specific personality

trait might be linked to job performance.

Another approach to improving prediction with personality assessment is to measure narrower traits that are

specific facets of the Big Five (Paunonen, Rothstein, & Jackson, 1999; Schneider, Hough, & Dunnette,

1996). Researchers who argue for this approach posit that some facets of a trait may relate to a particular job

performance dimension, but that others may not - or may even be negatively related. Measuring at the broad


level may dilute the relationship between the narrower facet and the job performance dimension. For

example, achievement and reliability are both narrow facets of conscientiousness. One might expect a strong

positive relationship between reliability and retention, but there may not be a relationship between

achievement and retention. Using an assessment that measures the broad trait of conscientiousness may

under-predict retention. Researchers conducted a meta-analysis and demonstrated the superiority of using

narrow facets to predict specific job performance dimensions (Woo, Chernyshenko, Stark, & Conz, 2014).

Furthermore, many personality assessments used for pre-employment selection are actually compound traits

– or constellations of narrow facets (Schneider et. al., 1996). These constellations may come from one or

more of the broader personality traits. In fact, instruments used in both the Hogan and Holland (2003) and

Bartram (2005) meta-analyses contain measures of compound traits.

Content Development

A review of the literature on personality in the workplace allowed us to identify several competencies, or

compound traits, that are related to successful job performance for many jobs. The ten scales, their

definitions, and example items are listed below. Please note, example items are similar to, but do not reflect,

actual items from the assessment.

Needs Structure

Often, following rules and procedures is critical to successful job performance. When employees do not

follow established organizational policies, work processes may be performed incorrectly, work products may

be flawed, customer service levels may suffer, the organization may become a victim to fraudulent wage and

expense billings, and in some cases the organization may be liable for the actions of the employee. The

Needs Structure scale was designed to evaluate an applicants’ tendency to adhere to organization rules and

procedures and should be important in jobs where rule-abiding behaviors are related to job performance.

Example: I prefer to make extensive plans before I start any project.

Innovative and Creative

For many positions, Innovation and Creativity is critical to successful job performance. Individuals with

high levels of this competency should be more likely to generate new products, develop improved methods

for producing work, and develop solutions to problems.

Example: Some of my suggestions are really eccentric.


Enjoys Problem-Solving

Enjoys Problem-Solving is a competency designed to predict which applicants will be more successful in

roles dealing with data, completing analyses, and conducting research. Individuals who enjoy problem

solving should be more successful in roles that require analytical thinking.

Example: I enjoy learning about how things work.

Competitive

The Competitive competency evaluates the extent to which in individual is likely to do what is necessary to

accomplish their goals – which may be of concern if the individual’s goals are not aligned with the

organization’s goals. Individuals who score highly on this competency are characterized as having concern

for outcomes rather than the feelings of coworkers.

Example: I am not above using people to get my way.

Competitive v2

The Competitive v2 competency evaluates the degree to which an individual is driven by a desire to achieve

objectives and outperform their peers. Competitiveness is the tendency to evaluate one’s performance in

comparison to others; a desire to do better than others, an enjoyment of the situations that can lead to a

clear winner; thrives in an environment where people are differentiated by accomplishments.

Unless otherwise noted, the results in this technical manual refer to Competitive scale from the original

AIMS test development. The Competitive v2 scale began being used in July 2016, and data is being collected.

The decision was made to replace the original Competitive scale due to customer complaints and upon

psychometric review of its continuing usefulness, based on research conducted through June, 2016.

Example: I get satisfaction from beating out the competition.

Seeks Perfection

When the quality of work (as opposed to pace) is important for success in a role, the Seeks Perfection

competency should predict performance. Those who score high on this competency are more likely to

double-check their work and have higher rates of accuracy. Additionally, individuals high on this

competency are more likely to be detail-oriented.

Example: My work tends to be faultless.


Develops Relationships

For many roles, working in teams and with others is critical for successful performance. The Develops

Relationships scale was designed to predict which individuals are more likely to develop productive working

relationships and be successful in a team situation.

Example: I have never deliberately said anything that hurt someone's feelings.

Expressive and Outgoing

Individuals who are Expressive and Outgoing are more likely to engage with others. Being Expressive and

Outgoing is also characterized by the ability to influence and persuade others. This competency is important

for success in roles where influence is necessary such as leadership and sales positions.

Example: I tend to take control in most work situations.

Corporate Citizenship

The Corporate Citizenship competency was designed to measure applicants’ tendency to conduct

themselves ethically and responsibly in an organization. Individuals who score highly on this competency are

likely to be honest with managers and colleagues and avoid taking advantage of people and/or situations

just because it suits their self-interest.

Example: I would never use manipulation as a tactic to advance my goals.

Exhibits a Positive Work Attitude

This competency evaluates the tendency for an individual to feel positively toward their job and the

organization they work for. Individuals who score highly on this competency are characterized as feeling

satisfied with their work and are more likely to put in extra effort when needed.

Example: I volunteer for additional work.

Adaptable

The Adaptable competency measures an individual’s tendency to adjust to changes in their work

environment. Individuals who score highly on this competency thrive in fast-paced settings and respond

well to variety.

Example: I think organizational changes are fun and exciting.


The figure below outlines the hypothesized relationships between the AIMs scales and the Big Five traits.

As mentioned above, the AIMs scales are compound traits and measure competencies related to multiple

narrow facets of personality and these facets may be under one or more of the five broader personality traits.

Therefore, an AIMs scale may be expected to relate, albeit moderately, to multiple Big Five traits.

Figure 3. Hypothesized Relationships between the AIMs Scales and the Big Five

The original assessment consisted of 100, Likert-type items. Each item consists of a single statement and

candidates are asked to indicate their level of agreement with the statement. Preliminary data analyses

allowed us to shorten the scales such that the assessment contains a total of 65 items. The AIMs assessment

is available in two forms. The professional form measures all ten competencies. The entry-level form

measures eight competencies as it was determined that Expressive and Outgoing and Innovative and

Creative would be less critical competencies for these roles. The time to take the professional form is

Emotional

Stability

Openness

Extraversion

Conscientiousnes

s

Agreeableness



Competitive

Seeks Perfection




Exhibits a Positive Work

Attitude

Adaptable

Needs Structure


estimated to be between 6 and 7 minutes, and the time to take the entry-level form is estimated to be 5 and

6 minutes. The estimates were calculated using a sample of 402 individuals.

Reliability

The descriptive statistics and reliability estimates of the AIMs scales were estimated using the data collected

via MTurk as described in the previous section. As can be seen from the table below, the AIMs scales have

acceptable levels of internal consistency.

Table 6. Descriptive Statistics and Reliability Evidence for the AIMS Scales

Competency N M SD Alpha

Needs Structure 675 26.76 4.85 .84

Innovative & Creative 567 25.08 5.33 .85

Enjoys Problem-Solving 675 26.47 5.67 .86

Competitive 675 13.27 6.15 .86

Seeks Perfection 675 25.66 5.46 .84

Develops Relationships 675 25.97 5.36 .83

Expressive & Outgoing 567 19.71 6.22 .82

Corporate Citizenship 675 45.33 7.73 .89

Exhibits a Positive Work Attitude 675 28.15 5.14 .80

Adaptable 675 25.49 4.57 .69

Competitive v2* 318 31.08 7.21 .85

*Competitive v2 replaced Competitive in July 2016

Validity

A Confirmatory Factor Analyses (CFA) was run using R (version 3.1.2) and the “sem” package to evaluate

whether or not the internal structure of the assessment conformed to the hypothesized structure. In other

words, we assessed the extent to which the items loaded onto the particular scale they were assigned to. The

Root Mean Square Error (RMSEA), an indicator of model fit, indicates that the model was satisfactory,

RMSEA=.0675, χ2 (1970, N=567) =7057.80, p<.001. This analysis provides validation support for the

internal structure of the assessment.

As mentioned previously, the content for the AIMs scales were modified and adapted from a longer

assessment that had been used in previous validation research. The developer of the original content

provided the results of several criterion-related validity studies and tables with these results can be found in

Appendix C.









Fairness


samples t-tests were performed to compare mean scores of the subgroups (Tables 7 -11). There were too

few cases (n<22) to run analyses for the American Indian or Alaska Native, Native Hawaiian or Other

Pacific Islander, Two or More Races, and Other subgroups. There were significant mean differences on the

AIMs scales between males and females. However, all differences were relatively small or they were in favor

of the focal subgroup, females. There were no significant mean differences based on ethnicity. There were a

few significant mean differences with respect to age group, but all but one of these were in favor of the

focal subgroup, 40 and older. There were a few significant mean differences between the Asian and White

subgroups. However, individuals in this sample were from multiple countries and it’s possible that these

differences might be reflections of cultural differences. Additional research is needed to further evaluate

these differences, to calculate country-specific norms, and to evaluate subgroup differences within country.

There were no significant differences between the Black or African American subgroup and the White

subgroup.

Table 7. Evaluation of AIMs Score Differences by Gender

Scale

Female Male n M SD n M SD t d

Needs Structure 287 27.19 4.94 380 26.42 4.77 -2.03* -0.16

Innovative and Creative 240 24.62 5.47 319 25.39 5.20 1.69 0.14

Enjoys Problem Solving 287 25.30 5.84 380 27.32 5.39 4.62* 0.36

Competitive 287 12.08 5.85 380 14.08 6.19 4.23* 0.33

Seeks Perfection 287 26.20 5.46 380 25.22 5.45 -2.30* -0.18

Develops Relationships 287 26.40 5.25 380 25.63 5.42 -1.84 -0.14

Expressive and Outgoing 240 18.67 6.05 319 20.38 6.18 3.27* 0.28

Corporate Citizenship 287 47.44 6.88 380 43.79 7.97 -6.21* -0.49

Exhibits a Positive Work Attitude 287 29.10 5.04 380 27.52 5.06 -4.00* -0.31

Adaptable 287 26.01 4.73 380 25.08 4.41 -2.61* -0.20

Note. *p<.05


Table 8. Evaluation of AIMs Score Differences by Ethnicity

Hispanic or

Latino Not Hispanic or

Latino

Scale n M SD n M SD t d

Needs Structure 46 27.52 4.67 574 26.62 4.87 -1.21 -0.19


Enjoys Problem Solving 46 26.74 5.98 574 26.51 5.63 -0.27 -0.04

Competitive 46 14.13 6.40 574 13.01 5.92 -1.23 -0.19

Seeks Perfection 46 26.33 5.61 574 25.56 5.46 -0.92 -0.14

Develops Relationships 46 25.72 5.61 574 25.97 5.37 0.30 0.05

Expressive and Outgoing 33 21.12 7.23 485 19.53 6.09 -1.43 -0.26

Corporate Citizenship 46 44.22 8.41 574 45.68 7.40 1.27 0.20

Exhibits a Positive Work Attitude 46 27.63 5.56 574 28.28 5.12 0.82 0.13

Adaptable 46 24.8 5.53 574 25.57 4.47 1.10 0.17

Note. *p<.05

Table 9. Evaluation of AIMs Score Differences by Age Group

40 and Older Less than 40


Needs Structure 150 27.57 5.36 524 26.54 4.67 -2.30* -0.21

Innovative and Creative 138 25.11 5.27 428 25.07 5.37 -0.08 -0.01

Enjoys Problem Solving 150 26.31 5.08 524 26.51 5.84 0.38 0.04

Competitive 150 10.38 4.02 524 14.08 6.40 6.71* 0.62

Seeks Perfection 150 25.48 6.07 524 25.72 5.28 0.47 0.04


Expressive and Outgoing 138 17.59 5.10 428 20.38 6.40 4.66* 0.46

Corporate Citizenship 150 48.42 5.37 524 44.45 8.08 -5.67* -0.52

Exhibits a Positive Work Attitude 150 29.79 4.83 524 27.69 5.14 -4.47* -0.41

Adaptable 150 26.25 4.63 524 25.27 4.53 -2.32* -0.22

Note. *p<.05


Table 10. Evaluation of AIMs Score Differences by Race Groups: Asian and White

Asian White


Needs Structure 89 27.25 4.86 518 26.66 4.80 -1.07 -0.12

Innovative and Creative 76 25.28 5.26 434 24.93 5.29 -0.53 -0.07


Competitive 89 18.35 7.80 518 12.38 5.31 -9.06* -1.04



Expressive and Outgoing 76 24.21 7.23 434 18.95 5.68 -7.13* -0.89

Corporate Citizenship 89 38.60 10.27 518 46.49 6.56 9.52* 1.09

Exhibits a Positive Work Attitude 89 25.75 5.25 518 28.55 5.00 4.84* 0.56

Adaptable 89 24.72 3.59 518 25.53 4.62 1.57 0.18

Note. *p<.05

Table 11. Evaluation of AIMs Score Differences by Race Groups: Black or African American and White

Black or African

American White


Needs Structure 25 26.76 5.76 518 26.66 4.08 -0.10 -0.02



Competitive 25 13.64 6.89 518 12.38 5.31 -1.14 -0.23


Develops Relationships 25 25.04 6.17 518 25.86 5.44 0.73 0.15

Expressive and Outgoing 19 17.68 5.65 434 18.95 5.68 0.95 0.22

Corporate Citizenship 25 45.40 7.31 518 46.49 6.56 0.81 0.17

Exhibits a Positive Work Attitude 25 27.92 6.05 518 28.55 5.00 0.61 0.12

Adaptable 25 25.88 6.06 518 25.53 4.62 -0.36 -0.07

Note. *p<.05


Behavioral History Survey

Previous Research

Past behavior often predicts future behavior. Biographical data, or bio-data, assessments contain items

developed by identifying patterns associated with high productivity and low turnover. Hunter and Hunter

(1984) report the average validity coefficient for bio-data assessments to be .38. Other researchers have

estimated the validity to be .35 for supervisors (Rothstein, Schmidt, Erwin, Owens, & Sparks, 1990) and .53

for managers (Carlson, Scullen, Schmidt, Rothstein, & Erwin, 1999).

For example, Rothstein, et al., (1990) identified several bio-data factors that could be applied to many jobs.

These included things like having a pervasive feeling of self-worth and confidence; believing that he or she

works better and faster than others in his or her area of specialization; having been recognized for

accomplishments; being outgoing; being a good communicator; taking clear positions; and, feeling healthy

and satisfied with current life situations. Rothstein screened each biographical item for cross-validity then

meta-analyzed 11,000 first-line supervisors from different organizations, age levels, genders, job experience

levels and tenures. He concluded that in all cases, validity estimates for these factors were generalizable,

stable across time, and did not appear to stem from acquired skills, knowledge or abilities.

McDaniel (1989) evaluated biographical questions about school suspensions, drug use, quitting school, prior

employment experience, grades, club memberships, contacts within the legal system, and socioeconomic

status. The results successfully predicted discharge from the military for problems such as alcohol and drug

use, desertion, imprisonment, and “discreditable incidents.”

Oswald, Schmitt, Kim, Ramsay, and Gillespie (2004) reported statistically significant bio-data correlations

with 12 dimensions of college student performance: knowledge, learning, artistic ability, multicultural

sensitivity, leadership, interpersonal skills, citizenship, health, careers, adaptability, perseverance, and ethics.

Their work showed incremental validity over the traditional use of SAT and ACT with fewer differences

between subgroups than traditional admission measures.

In a similar vein, Kanfer, Crosby, and Brandt (1988) identified correlations between bio-data and tenure; and

a study of 555 real estate agents, Klimoski and Childs (1986) identified five major bio-data factors associated

with job, personal and career success. They included social orientation, economic stability, work ethic

orientation, educational achievement and interpersonal confidence.

Content Development

The Behavioral History Survey consists of three biographical history areas that generalize across most jobs:

tenure, performance, and unproductive behavior. Questions and forms were developed for both

professional and entry-level positions using a panel of experienced managers. Each bio-data item was


reviewed by an expert management panel and scored using a modified Angoff method. There are a few

versions of each form available because some of the items were relevant for some positions but irrelevant

for others. For example, an item asking about experience with an industry would not be appropriate for a

position that spans multiple industries (e.g. Administrative Assistant). Scores for each competency are

calculated by averaging across items within that competency. The use of averaging, rather than summing

across items, allows HR Avatar to compare similar scales across multiple versions of the assessment. The

entry-level form contains 14-15 items and takes approximately 2 minutes to complete. The professional

form contains 18-20 items and takes approximately 6 minutes to complete. The estimates were estimated

with two samples (N= 70 and N=371, respectively).

Performance

The Performance scale asks questions related to past performance on the job and should predict multiple

dimensions of job performance.

Example: How many times have you been promoted at work?

Tenure

The Tenure scale asks questions specifically related to tenure on previous jobs. This scale should predict

retention.

Example: What is your longest tenure with an organization?

Unproductive Behavior

The Unproductive scale includes items related to behavior that is disruptive and violates rules and/or social

norms.

Example: In the past, how many times have you violated a company policy?

Reliability

The scales in the Behavioral History Survey are relatively short – some with as few as three items.

Additionally, the scales were not expected to be internally consistent as there is no evidence that the

individual item responses should be highly correlated with one another. Therefore, a test-retest reliability

estimate was determined to be more appropriate than one of internal consistency (e.g., Cronbach’s alpha).

Descriptive statistics and reliability estimates were calculated using data collected via MTurk. This data

collection effort is summarized in the Cognitive Work Simulations section. Because participants were

allowed to take multiple versions of the solution, several participants took the Behavioral History Surveys


more than once. Test-retest reliability estimates were calculated by correlating the scale scores between the

first and second administrations of the form.

For the Professional form, the sample was restricted to those individuals who had at least two days between

administrations. The largest number of days between administrations was 25. On average, there were 11

days between administrations (SD=7.49). The descriptive statistics and test-retest reliability estimates are in

Table 12 and all reliability estimates are acceptable.

For the Entry-Level form, the sample was restricted to those individuals who had at least two days, but less

than 30 days between administrations. The largest number of days between administrations was 20. On

average, there were 7 days between administrations (SD=4.76). The descriptive statistics and test-retest

reliability estimates are in Table 13 and all reliability estimates are acceptable.

Please note, there are multiple versions of the Performance scale for both the Professional and the Entry-

Level forms as described above. The descriptive statistics, test-retest reliability estimates and fairness

analyses for this scale are estimated by averaging across various versions of the scales.

Table 12. Descriptive Statistics and Reliability Evidence for the Professional Behavioral History Survey

Competency N M SD ρxx N for

ρxx

Performance 303 2.34 0.29 .81 65

Tenure 303 2.29 0.40 .71 65

Unproductive Behavior 303 2.62 0.42 .72 65

Table 13. Descriptive Statistics and Reliability Evidence for the Entry-Level Behavioral History Survey

Competency N M SD ρxx N for

ρxx

Performance 593 2.43 0.38 .74 151

Tenure 593 2.62 0.36 .71 151

Unproductive Behavior 593 2.59 0.42 .74 151

Fairness


samples t-tests were performed to compare mean scores of the subgroups (Tables 14-17). There were too

few cases (n<25) to run analyses for the American Indian or Alaska Native, Black or African American,

Native Hawaiian or Other Pacific Islander, Two or More Races, and Other subgroups. There were

significant mean differences on the Behavioral History scales between males and females. However, all


differences were relatively small or they were in favor of the focal subgroup, females. There were a few

significant mean differences based on ethnicity. Unfortunately, the sample is too small for the Hispanic and

Latino subgroup to draw any conclusions. As data become available, further research will explore these

differences at the item and response level. There were significant mean differences with respect to age group,

but all were in favor of the focal subgroup, 40 and older. There were a few significant mean differences

between the Asian and White subgroups. However, individuals in this sample were from multiple countries

and it’s possible that these differences might be reflections of cultural differences. Additional research is

needed to further evaluate these differences, to calculate country-specific norms, and to evaluate subgroup

differences within country.

Table 14. Evaluation of Biographical History Survey Score Differences by Gender

Female Male


Professional Performance 123 2.33 0.29 177 2.34 0.30 0.00 0.03

Professional Tenure 123 2.36 0.42 177 2.24 038 -2.58* -0.30

Professional Unproductive Behavior 123 2.71 0.35 177 2.57 0.45 -2.89* -0.34

Entry-Level Performance 240 2.33 0.37 305 2.34 0.36 0.21 0.02

Entry-Level Tenure 240 2.66 0.37 305 2.59 0.37 -2.19* -0.19

Entry-Level Unproductive Behavior 240 2.68 0.36 305 2.54 0.45 -3.93* -0.34

Note. *p<.05


Table 15. Evaluation of Biographical History Survey Score Differences by Ethnicity

Hispanic or Latino

Not Hispanic or

Latino


Professional Performance 25 2.36 0.30 260 2.34 0.29 -0.25 -0.05

Professional Tenure 25 2.13 0.44 260 2.31 0.39 2.18* 0.46

Professional Unproductive Behavior 25 2.61 0.43 260 2.62 0.41 0.12 0.02

Entry-Level Performance 38 2.13 0.41 467 2.33 0.35 3.37* 0.57

Entry-Level Tenure 38 2.57 0.39 467 2.64 0.35 1.18 0.20

Entry-Level Unproductive Behavior 38 2.58 0.37 467 2.61 0.40 0.45 0.08

Note. *p<.05

Table 16. Evaluation of Biographical History Survey Score Differences by Age Group

40 and Older Less than 40


Professional Performance 60 2.44 0.26 243 2.31 0.30 -3.14* -0.45

Professional Tenure 60 2.49 0.31 243 2.25 0.40 -4.34* -0.62

Professional Unproductive Behavior 60 2.70 0.34 243 2.61 0.43 -1.51 -0.22

Entry-Level Performance 114 2.41 0.30 439 2.31 0.38 -2.52* -0.26

Entry-Level Tenure 114 2.82 0.24 439 2.57 0.38 -6.68* -0.70

Entry-Level Unproductive Behavior 114 2.67 0.35 439 2.58 0.43 -2.06* -0.22

Note. *p<.05

Table 17. Evaluation of Biographical History Survey Score Differences by Race Groups: Asian and White

Asian White


Professional Performance 27 2.34 0.34 243 2.34 0.29 0.08 0.02

Professional Tenure 27 2.07 0.50 243 2.35 0.37 3.59* 0.73

Professional Unproductive Behavior 27 2.51 0.49 243 2.64 0.41 1.53 0.31

Entry-Level Performance 77 2.27 0.38 418 2.34 0.35 1.44 0.18

Entry-Level Tenure 77 2.45 0.40 418 2.67 0.34 5.07* 0.63

Entry-Level Unproductive Behavior 77 2.48 0.51 418 2.62 0.39 2.75* 0.34

Note. *p<.05

Knowledge and Skills Tests

Development Overview

Candidates need more than just the right combination of abilities, personality characteristics and

background in order to be successful in a job. Most often, specific knowledge and/or specific skills are also

required. Job knowledge tests measure job-relelevant declaritive knowledge such as technical information,


standards, and best practices as well as knowledge of specific processess and procedures. Job knowledge

tests serve as an indicator of previous job performance and serve as proximal predictors of future job

performance. Skills assessments evaluate a candidate’s ability to perform a specific task. Candidiates who

begin work with a sufficient level of knowledge and/or skill should require less training and beable to

perform better faster. Hunter and Hunter (1984) report the average validity coefficient for job knowledge

tests to be .48. In their meta-analysis, Dye, Reck, and McDaniel (2007) found the average corrected

correlation coefficient for job knowledge tests to be .45 for job performance and .47 for training.

Correlations were even higher for complex jobs and higher to the extent that the job knowledge test was

similar to the job.

HR Avatar has developed dozens of knowledge assessments. See Appendix A for a complete listing. The

knowledge assessments were developed by reviewing multiple resources to identify appropriate items. For

example, The Food Safety Fundamentals assessment was developed by reviewing, among other resources,

the USDA Safe Food Handling Fact Sheets. Each knowledge test contains an item bank and half of the

items are randomly selected to be administered to an applicant. Research on the validity, reliability, and

fairness of these assessments is forthcoming. HR Avatar has three skills assessments: Sales Situation

Analysis, Typing Test, and Essay Test. The development and research on these assessments is described in

more detail below.

Validity

Initial validation evidence is available, in a study with 130 managers, which demonstrates the validity of the

assessments. The assessment demonstrated that it predicts job performance (r=.25; p<.01), using a simple

“High” or “Marginal” categorization of job performance. We expect validity evidence to be more robust in

future studies with better criterion measures. A brief summary of the results is presented in Appendix G.

Sales Situation Analysis

Test Development

The Sales Situation Analysis is a scale that is integrated into the Business Sales Cognitive Work Simulation

(described above). In the simulation, the candidate is asked to assist and respond to customers and

colleagues and solve business-related problems. The Sales Situation Analysis scale specifically evaluates a

candidate’s ability to understand a customer’s needs and identify the most appropriate follow-up actions. In

the simulation, the candidate must read an email communication from a customer and then identify the

customer’s primary concern. Next, the candidate must identify which of several action items are most

appropriate for the specific sales situation.


Reliability

The descriptive statistics and reliability estimate for the Sales Situation Analysis were estimated using the

data collected via MTurk as described in the previous section. Table 18 contains the descriptive statistics and

reliability estimate for this scale. These analyses led the developers to make substantial edits to these items

and we anticipate that edits will substantially improve the reliability of the tool. Further research evaluating

the effectiveness of these changes is forthcoming.


Table 18. Descriptive Statistics and Reliability Evidence for the Sales Situation Analysis Assessment

# of

Items N M SD Alpha

Sales Situation Analysis 6 98 3.72 1.04 .22

Fairness

In order to evaluate how different demographic subgroups perform on the assessment, independent

samples t-tests were performed to compare mean scores of the subgroups. The results are provided in Table

19. Although data were collected on ethnic group, the sample size for the ‘Hispanic or Latino’ subgroup was

too small for analyses (n<15). Data were also collected for racial group. However, the sample sizes for all

but the “White” subgroup were too small for analysis (n<30). There were not significant mean differences

based on gender or age group.

Table 19. Evaluation of Sales Situation Analysis Score Differences

n M SD t d

Female 41 3.73 1.06 .09 .02

Male 56 3.71 1.05

40 and older 17 4.00 1.08 1.19 .31

Less than 40 81 3.67 1.03

Note. *p<.05

Typing Test

Test Development

The typing test consists of three typing tasks. For each task, applicants are asked to type a short passage.

Each typing task is randomly selected from a group of five passages (15 total passages). For each task, the

words per minute are calculated. This score is modified to an accuracy-adjusted words per minute by

calculating and factoring in the rate of typing errors. The typing scores are calculated by averaging the words

per minute and accuracy-adjusted words per minute across the three tasks.

Reliability

The typing test was piloted via MTurk (N=155). The majority of the sample was from the US (92.9%). The

pilot assessment consisted of five typing tasks and each typing task was randomly selected from a group of

five passages (25 total passages). Five one-way analyses of variance were completed to evaluate differences

in scores across the passages available for each item. No significant differences were found, F(4, 150)=.56,

p=.70, F(4, 150)=.27, p=.89, F(4, 150)=.79, p=.54, F(2, 121)=1.12, p=.33, F(2, 121)=.22, p=.80. Descriptive

statistics and the alpha estimate of internal consistency were estimated using the first three items from the


five item pilot assessment (see Table 20). The internal consistency of the assessment is well above accepted

standards.

Table 20. Descriptive Statistics and Reliability Estimates for the Typing Test

Typing Test N M SD alpha

3 Items 155 46.92 15.63 .97

Fairness


samples t-tests were performed to compare mean scores of the subgroups (Table 21). Although data were

collected on ethnic group, the sample size for the ‘Hispanic or Latino’ subgroup was too small for analyses

(n=11). There were too few cases (n<10) to run analyses for the African American or Black, American

Indian or Alaska Native, Native Hawaiian or Other Pacific Islander, Two or More Races, and Other

subgroups. Significant mean differences were found for gender and between the Asian and White racial

subgroups. Given that this sample consisted of people piloting the assessment but not necessarily taking the

assessment in order to obtain a job, it is possible that the individuals were not attempting to perform their

best. Further research will evaluate norms using an applicant sample.

Table 21. Evaluation of Essay Score Differences by Subgroup

n M SD t d

Female 69 43.64 15.12 2.59* .42

Male 83 50.09 15.39

40 and Older 33 44.20 11.51 1.19 .26

Less than 40 120 47.85 16.50

Asian 26 41.76 18.23 2.28* .46

White 108 49.45 14.68

Note. *p<.05

Essay Test

Test Development

Written communication is a key skill in many positions. Communicating via email, writing reports, and

creating presentations all require the ability to communicate effectively. The HR Avatar Essay Test is a fast

method for evaluating an applicant’s written communication skills. The HR Avatar Essay Test consists of

one of two writing prompts. The writing prompts were designed to be general enough to provide an

opportunity for anyone to be able to write a short essay. The writing prompts are included below.


1. Describe the pros and cons of working from home.

2. Describe the pros and cons of living in a big city.

Applicants are asked to write a short essay with a minimum of 100 words and are given an unlimited time to

do so.

The essays are scored using Discern, an open source, machine learning program. Discern was designed by

edX, a nonprofit organization founded by Harvard and the Massachusetts Institute of Technology (MIT)

(edX, 2015; Markoff, 2013). A YouTube video was published and provides some additional information on

how the program works: https://www.youtube.com/watch?v=zFeP678054U (Paruchuri, 2015). The system

produces a score that ranges from 0 to 100. A confidence estimate for the score is also computed. The

confidence estimate can range from 0 to 1. Scores with confidence estimates less than .10 are not considered

valid.

In order to calibrate the program, HR Avatar used MTurk to collect writing samples for the two writing

prompts (N=170 and N=163). The essays were scored on three areas: Grammar, Structure and Content, by

three independent raters. Prior to rating each essay, the raters were provided with scoring rubrics (Appendix

D) and training (Appendix E) for how to score the essays using the rubrics. A total score was calculated for

each essay by aggregating scores on Grammar, Structure, and Content and then averaging across the raters

and linearly transforming the scores to a scale of 0-100. Scores were entered into the program as the

calibration sample for Discern.

In order to evaluate the reliability of the ratings provided by the three raters, intra-class correlation

coefficients were calculated using a two-way, mixed effects model (ICC3) (Shrout & Fleiss, 1979) . The

ICC3 was chosen because it is the reliability of the average rating made by the specific raters in this study

that were of interest, and it is not necessary to generalize the reliability estimate to the population of

potential raters. There were ratings available for all three raters for 317 essays and the reliability of the

average aggregate rating was acceptable (ICC (3,3)=.74). There was also a large relationship between scores

on the two essay prompts, r(152)=.62, p<.01.

In early January 2015 additional improvements were made to the Essay scoring. First, the 196 essays that

were completed since the initial rollout were manually scored by an individual rater and re-entered into the

Discern program to further calibrate the system. Second, HR Avatar added some additional safeguards to

the automated scoring engine. The essays are truncated to 800 words. Essays with fewer than 100 words,

consisting of more than 25% of spelling errors, or more than 25% of grammar errors or style errors are

given a score of 0. An additional program was written to search the HR Avatar database for matching essay

content. The system uses a combination of three methods to determine the similarity between the essay and

existing content in the system as a way of detecting plagiarism: The Levenshtein Distance Strategy (Navarro,


2001), the Jaro distance metric (Jaro, 1989; Jaro, 1995), and the Jaro-Winkler distance metric (Winkler, 1990).

If matching content is discovered, it is assumed that the essay is plagiarized and applicants will receive a

score of 0.

Descriptives

HR Avatar has collected essays from 206 individuals who piloted the system between November 27th, 2014

and February 26th, 2015. Many of these data points are individuals piloting the system rather than applicants

applying for a job. However, based on an examination of the data, there appear to be a substantial number

of instances where it appears as if people were trying to perform well on the assessment. Therefore,

descriptive statistics have been provided in Table 22 for the essays after removing those cases in which a

“real” essay was not written (n=5). Please note that there were not significant score differences based on

which prompt was received, t(200)=1.18, p=.24, two-tailed, d-=17. The majority of the sample came from

the US and the Philippines, therefore descriptive statistics are provided at the country level for these

countries.

Table 22. Descriptive Statistics for Essay Scores

n M SD

Score 201 46.14 14.05

Confidence 201 0.73 0.14

Prompt 1 (Working from home) 103 44.99 13.00

Prompt 2 (Living in a big city) 99 47.32 15.03

Philippines 108 43.31 13.85

US 73 50.15 13.05

Fairness


samples t-tests were performed to compare mean scores of the subgroups. Although data were collected on

ethnic group, the sample size for the ‘Hispanic or Latino’ subgroup was too small for analyses (n=11). There

were too few cases (n<10) to run analyses for the American Indian or Alaska Native, Native Hawaiian or

Other Pacific Islander, Two or More Races, and Other subgroups. Both the Asian and the Black or African

American subgroups were compared to the White subgroup, most often considered the reference group. As

can be seen in the table below, the only significant differences between scores were between the Asian and

The White subgroups. This finding may be related to differences found within the two primary countries

samples: The US and the Philippines. Further analyses will be completed once data allows and score

differences between racial subgroups will be estimated within country.


Table 23. Evaluation of Essay Score Differences by Subgroup

n M SD t d

Female 113 45.85 13.33 .26 .04

Male 80 46.38 15.31

40 and Older 67 47.00 12.14 .58 .09

Less than 40 133 45.79 14.80

Asian 103 42.99 13.89 2.95* .49

Black or African American 25 46.88 13.64 .87 .21

White 56 49.70 13.40

Note. *p<.05

Validity








Solution Scoring

In addition to providing scores at the scale level, HR Avatar also provides an overall score as an indication

of overall fit between the candidate and a given job. All competencies are grouped within four broad

categories: Cognitive Abilities, Skills and Knowledge, AIMs, and Behavioral History. Each competency, or

scale, is converted to a z-score. For the Cognitive Abilities, Skills and Knowledge, and Behavioral History

categories, a category score is created by averaging z-scores within each category.

For the AIMs and Cognitive Ability categories, O*NET is used to determine which of the competencies are

relevant for a given job and to determine the appropriate weights. O*NET is an online database that

contains specific information about hundreds of occupations (Peterson, et al., 2001). The information

gathered about each job is categorized using strongly supported theoretical models about behavior in the

workplace. Additionally, the process for gathering the information and documenting it is a collaborative

effort. The O*NET Skills and Abilities Importance ratings are the average importance ratings given by at

least eight occupational analysts (Fleisher & Tsacoumis, 2012a, 2012b). The Occupational Analysts all have

two or more years of work experience, two or more years of graduate level education in a program related to


human resources or workplace psychology, and coursework in research methods and job analysis.

Occupational Analysts are provided with extensive training and detailed information about each job prior to

making ratings including job description, knowledge requirements, task descriptions and work context. The

AIMs and Cognitive scales were mapped onto the O*NET Worker Characteristics and Worker

Requirements. Weights are applied to the scales such that each scale receives a weight that is equivalent to

the proportion of its importance rating within O*NET. When a competency is listed more than once, which

is sometimes the case given that several Worker Characteristics or Worker Requirements might be mapped

to a given competency, the weight given to the competency corresponds to the highest importance rating.

For the knowledge tests, the raw scores are the percent correct. As of yet, these assessments do not have

sufficient data to estimate stable normative parameters. Therefore, a mean of .70 and an SD of .25 will be

used to estimate z-scores for applicants.

An overall z-score is calculated by computing a weighted average of the competency categories. The

following weights are assigned to each category:

Cognitive Ability competencies = 1

Skills/Knowledge competencies = 0.8

AIMs competencies = 0.7

Behavioral History competencies = 0.4

The overall z-score is transformed to a Normal Curve Equivalent (NCE) score. NCE scores have a mean of

50 and a standard deviation of 21.06 and maintain their equal-interval properties.

Construct Validity Evidence

Tables 24-30 contain the correlations between the scales in the HR Avatar Assessment Solution. Generally

speaking, the relationships conform to what would be expected. For example, within the AIMs assessment,

there is a strong negative relationship between the Competitive scales and the Exhibits a Positive Work

Attitude scale. The Enjoys the Problem Solving and Innovative and Creative scales are highly correlated

which is to be expected as they are both facets of the Openness trait of the Big Five. The relationships

between the Behavioral History scales and AIMs scales also conform to expectations. Specifically, the

Behavioral History Scales are negatively correlated with the Competitive scale and positively correlated with

Exhibits a Positive Work Attitude and Corporate Citizenship scales. Additionally, Enjoys Problem Solving is

strongly correlated with the Performance scale of the Behavioral History Survey.


Table 24. Correlations between the AIMs Scales

Exhibits a Positive

Work Attitude


Competitive Expressive

and Outgoing

Innovative and

Creative

Seeks Perfection



Needs Structure


.68** 675

Competitive -.67**

675 -.85**

675


-.23** 567

-.50** 567

.49** 567


.25** 567

.08 567

-.05 567

.41** 567

Seeks Perfection

.11** 675

.14** 675

-.08* 675

.12** 567

.40** 567


.36** 675

.24** 675

-.24** 675

.38** 567

.68** 567

.43** 675


.26** 675

.13** 675

-.13** 675

.33** 567

.72** 567

.43** 675

.54** 675

Needs Structure

.31** 675

.30** 675

-.30** 675

.00 567

.35** 567

.64** 675

.50** 675

.42** 675

Adaptability .52**

675 .39**

675 -.39**

675 .05

567 .55**

567 .28**

675 .61**

675 .49** 675

.32 675

Note. *p<.05, **p<.01


Table 25. Correlations between the Professional Behavioral History Scales

Performance Tenure

Tenure .17**

303

Unproductive Behavior .02

303 .09

303

Note. *p<.05, **p<.01

Table 26. Correlations between the Entry-Level Behavioral History Scales

Performance Tenure

Tenure .14**

510

Unproductive Behavior .05

510 .21**

553

Note. *p<.05, **p<.01


Table 27. Correlations between HR Avatar AIMs and Behavioral History Scales

Professional Entry-Level

Performance Tenure Unproductive Behavior



.34** 513

.11** 513

.27** 1682

.34** 1069

.11** 1169

.33** 1169

Corporate Citizenship .27**

513 .22**

513 .35**

1682 .23**

1069 .20** 1169

.38** 1169

Competitive -.25**

513 -.23** 513

-.35** 1682

-.22** 1069

-.21** 1169

-.39** 1169

Expressive and Outgoing .24**

513 -.05

513 -.14**

1460 .25**

947 -.15**

947 -.17**

947

Innovative and Creative .42**

513 .00

513 .06*

1460 .55**

1069 -.06 947

.07* 947

Seeks Perfection .23**

513 .10*

513 .17**

1682 .28**

947 -.01

1169 .15**

1169

Develops Relationships .43**

513 .12**

513 .13**

1682 .47**

1069 .01

1169 .11**

1169

Enjoys Problem-Solving .48**

513 .05

513 .03

1682 .57**

1069 -.01

1169 .06

1169

Needs Structure .15**

513 .18**

513 .31**

1682 .24**

1069 .04

1169 .25**

1169

Adaptability .35**

513 .07

513 .13**

1682 .39**

1069 .02

1169 .15**

1169

Note. *p<.05, **p<.01


Table 28. Correlations between the Two Cognitive Workplace Simulation Scales: Attention to Detail and Analytical Thinking

Simulation Correlation

/N

Administration .28**

119

Business/Finance .26**

121


.52** 137

Remote Customer Service .59**

98


.48** 116

General Office Workplace .28**

99

Entry-Level Office .57**

142

Information Technology .54**

98

Manager .42**

97

Retail Sales .47**

60

Business Sales .48**

94

First-Line Supervisor .43**

102

Technician .54**

114

Basic Entry-Level .51**

171

Note. *p<.05, **p<.01


Table 29. Correlations between the AIMs and the Cognitive Workplace Simulation Scales


Scale Exhibits a

Positive Work Attitude


Competitive Expressive and

Outgoing

Innovative and

Creative

Seeks Perfection



Needs Structure

Adaptability

Administration

Analytical Thinking

-.04 120

.13 120

-.11 120

-.04 120

-.14 120

-.10 120

-.11 120

.12 120

-.05 120

-.05 120

Attention to Detail

.13 119

.37** 119

14

-.40** 119

-.19* 119

-.07 119

.01 119

.05 119

.14 119

.18* 119

.01 119


Attention to Detail

.14 100

.29** 100

-.24* 100

-.14 100

.04 100

.17 100

.00 100

.07 100

.11 100

.01 100

Business/ Finance

Analytical Thinking

.02 122

.11 122

-.11 122

NA NA -.27** 122

-.11 122

-.01 122

-.23** 122

-.02 122

Attention to Detail

.20* 121

.33** 121

-.33** 121

NA NA .09

121 -.12

121 .04

121 .12

121 .00

121


Analytical Thinking

.16 139

.25** 139

-.28** 139

-.17 139

.03 139

-.09 139

.03 139

.15 139

-.11 139

.05 139

Attention to Detail

.31** 137

.40** 137

-.49** 137

-.17* 137

.05 137

.04 137

.09 137

.10 137

.09 137

.13 137


Analytical Thinking

.22* 98

.26** 98

-.31** 98

-.11 98

.00 98

-.14 98

.02 98

.21* 98

-.13 98

.06 98

Attention to Detail

.24* 95

.35** 95

-.31** 95

-.12 95

-.02 95

-.09 95

.05 95

.06 95

-.06 95

.08 95


Analytical Thinking

-.02 116

.17 116

-.14 116

.01 116

-.06 116

-.04 116

-.11 116

.01 116

.02 116

-.04 116

Attention to Detail

.06 116

.33** 116

-.38** 116

-.13 116

.08 116

.15 116

.01 116

.11 116

.12 116

.07 116


Analytical Thinking

.03 100

.16 100

-.15 100

NA NA 100

.03 100

-.01 100

.33** 100

.03 100

.14 100

Attention to Detail

.20* 99

.33** 99

-.31** 99

NA NA -.05

99 .02

99 .09

99 .09

99 .14

99



Scale Exhibits a Positive Work Attitude


Competitive Expressive and

Outgoing

Innovative and

Creative

Seeks Perfection



Needs Structure

Adaptability

Entry-Level Office

Analytical Thinking

.28** 142

.41** 142

-.47** 142

-.02 142

.08 142

.08 142

.17 142

.25** 142

.10 142

.23* 142

Attention to Detail

.21* 142

.30** 142

-.34** 142

-.08 142

.04 142

-.01 142

.05 142

.18* 142

.00 142

.09 142


Analytical Thinking

.08 98

.05 98

-.10 98

-.01 98

.11 98

-.13 98

.00 98

.15 98

-.21* 98

.06 98

Attention to Detail

-.01 98

.07 98

-.12 98

-.07 98

.03 98

.02 98

-.07 98

.07 98

-.15 98

-.01 98

Manager

Analytical Thinking

.19 98

.07 98

-.11 98

-.02 98

.04 98

-.04 98

.05 98

.12 98

-.07 98

.00 98

Attention to Detail

.05 97

.07 97

-.14 97

-.02 97

-.02 97

-.08 97

.04 97

.08 97

-.08 97

-.05 97

Retail Sales

Analytical Thinking

.06 60

.27* 60

-.28* 60

-.16 60

.01 60

-.30* 60

-.12 60

.00 60

-.29* 60

.07 60

Attention to Detail

.22 60

.33** 60

-.36** 60

-.32 60

-.04 60

.05 60

-.05 60

.06 60

.01 60

-.10 60

Business Sales

Analytical Thinking

.04 98

.25* 98

-.18 98

-.19 98

-.20* 98

.00 98

-.06 98

.01 98

-.07 98

-.18 98

Attention to Detail

.07 94

.47** 94

-.39** 94

-.09 94

-.14 94

.06 94

.12 94

.04 94

.09 94

.02 94

First-Line Supervisor

Analytical Thinking

.27** 102

.35** 102

-.33** 102

-.06 102

.10 102

.07 102

.05 102

.27** 102

.02 102

.04 102

Attention to Detail

.21* 102

.42** 102

-.40** 102

-.20* 102

-.06 102

-.10 102

-.05 102

.03 102

-.11 102

-.01 102

Technician

Analytical Thinking

.20* 114

.38** 114

-.36** 114

-.25** 114

.15 114

.02 114

.05 114

.26** 114

.06 114

.07 114

Attention to Detail

.21* 114

.39** 114

-.36** 114

-.35** 114

-.02 114

.03 114

-.09 114

.08 114

-.02 114

-.07 114

Basic Entry-Level

Analytical Thinking

.01 171

.11 171

-.14 171

-.11 171

-.12 171

-.05 171

-.16* 171

-.13 171

-.11 171

-.09 171

Attention to Detail

.17* 171

.24** 171

-.26** 171

-.06 171

-.04 171

-.02 171

-.04 171

-.03 171

.01 171

.02 171


Table 30. Correlations between the Behavioral History Survey and the Cognitive Workplace Simulation Scales



Scale Performance Tenure Unproductive Behavior


Administration

Analytical Thinking

.15 120

.01 120

-.11 120

NA NA NA

Attention to Detail

.23* 119

.19* 119

.25** 119

NA NA NA


Attention to Detail

NA NA .24*

100 .20*

100 .11

100 .17

100

Business/Finance

Analytical Thinking

NA NA .10

122 -.02

122 .16

122 .17

122

Attention to Detail

NA NA .14

121 .08

121 .10

121 .17

121


Analytical Thinking

NA NA .01

139 .19*

139 .15

139 .11

139

Attention to Detail

NA NA .22*

137 .26**

137 .12

137 .30**

137


Analytical Thinking

.23* 98

.27** 98

.16 98

NA NA NA

Attention to Detail

.21* 98

.28** 98

.20* 98

NA NA NA


Analytical Thinking

NA NA -.02

116 .03

116 .08

116 .00

116

Attention to Detail

NA NA .15 .09

116 .05

116 .13

116


Analytical Thinking

NA NA .03

100 .17

100 .01

100 .11

100

Attention to Detail

NA NA .07

99 .19

99 .25*

99 .17

99

Entry-Level Office

Analytical Thinking

NA NA .11

142 .30**

142 .04

142 .10

142

Attention to Detail

NA NA .03

142 .21*

142 .13

142 .06

142

Information Analytical .17 -.09 .03 NA NA NA




Scale Performance Tenure Unproductive Behavior


Technology Thinking 98 98 98

Attention to Detail

.06 98

-.05 98

-.01 98

NA NA NA

Manager

Analytical Thinking

.19 98

.05 98

-.01 98

NA NA NA

Attention to Detail

.17 97

.18 97

.08 97

NA NA NA

Retail Sales

Analytical Thinking

NA NA .08

60 .10

60 .01

60 .13

60

Attention to Detail

NA NA .16

60 -.03

60 .15

60 .21

60

Business Sales

Analytical Thinking

.16 98

.26 98

.07 98

NA NA NA

Attention to Detail

.21 94

.18 94

.20 94

NA NA NA

First-Line Supervisor

Analytical Thinking

NA NA .10

102 .29**

102 -.07

102 .17

102

Attention to Detail

NA NA .12

102 .17

102 .13

102 .25*

102

Technician

Analytical Thinking

NA NA .04

114 .32**

114 .13

114 .08

114

Attention to Detail

NA NA .04

114 .10

114 .20*

114 .09

114

Basic Entry-Level

Analytical Thinking

NA NA .25

171 -.07

171 .15*

171 .06

171

Attention to Detail

NA NA .07

171 .07

171 .23**

171 .13

171

Note. *p<.05, **p<.01


Technical Requirements

The HR Avatar solution is designed to be taken on a personal computer or a mobile device

including tablets and mobile phones. A high bandwidth connection is recommended, but not

required. All HR Avatar videos are compressed to less than 500kbps. Lower bitrate versions are

used for mobile devices.

The following web browsers are supported:

Internet Explorer 6 and above with Flash 9.1.115 or above

Internet Explorer 9 and above without Flash

Chrome

Firefox

Opera

Safari

The Flesch-Kincaid Reading Grade Level score is estimated to be 5.8. This indicates that an

applicant must have a reading level similar to that of a 5th or 6th grader in order to comprehend the

text in the assessment.

HR Avatar recommends that the applicant take the assessment in a quiet setting that is free from

distractions. This will allow the applicant the best opportunity for demonstrating their skills and

abilities.

Future Research

HR Avatar is committed to providing employers with high quality and legally defensible assessments

for hiring employees. To that end, HR Avatar plans to continue accumulating reliability, validity, and

fairness evidence to support the use of the solutions. The list below contains several items from our

research agenda. Please contact us, if you have any interest in partnering with HR Avatar on any of

the projects below.

Reliability

Establish the internal consistency of the knowledge tests.

Establish the test-retest reliability of the knowledge tests, the Typing Test and the Essay Test.

Establish the test-retest reliability of the composite scores for each solution.


Validity

Conduct studies to establish the content-related validity of the knowledge assessments,

Typing Test and Essay Test.

Conduct a study to examine the convergent validity evidence for the Cognitive Work

Simulation by comparing scores on the Cognitive Work Simulation and other established

measures of cognitive ability.

Conduct a study to examine the convergent and divergent validity evidence for the AIMS

assessment by comparing scores on the AIMS assessment with other established measures of

personality – particularly those that measure the Big Five. The hypothesized relationships

can be found in Figure 1.

Accumulate additional criterion-related validity evidence at the individual assessment level and the

composite level by conducting multiple studies on each solution. Using these studies, conduct meta-

analyses to provide evidence for validity generalization. Validation evidence is available in two

studies that demonstrate the validity of the assessments. The first is a study with 130 managers and

the second is a study with 64 managers. In the first study, the assessment demonstrated that it

predicts job performance (r=.25; p<.01), using a simple “High” or “Marginal” categorization of job

performance, and a second study demonstrated the assessment predicted performance on a 100

point administrative performance appraisal, using a Spearman correlation (r=.25; p<.05). We expect

validity evidence to be more robust in future studies with better criterion measures. A brief summary


Fairness

Conduct a sensitivity review of all assessment content.

Conduct Differential Item Functioning (DIF) analyses for the items in the assessments to

determine if any of the items behave differently for subgroups as defined by race, ethnicity,

gender, and age group.

Evaluate mean score differences on each assessment and at the composite level by subgroup.

Simulate selection ratios for each group at various passing rates to estimate adverse impact

ratios.

Norms

Update estimates of global norms and estimate norms at the country and/or region level.

Compare scores across formats (PCs, tablets, mobile phones).


References

American Educational Research Association, American Psychological Association, National Council

on Measurement in Education. (2014). The Standards for Educational and Psychological Testing.

Washington DC: American Educational Research Association.

Barrick, M. R., & Mount, M. K. (1991). The big five personality dimensions and job performance: A

meta-analysis. Personnel Psychology, 44, 1-26.

Bartram, D. (2005). The great eight competencies: A criterion-centric approach to validation. Journal

of Applied Psychology, 90, 1185-1203.

Bertua, C., Anderson, N., & Salgado, J. F. (2005). The predeictive validity of cognitive ability tests: A

UK meta-analysis. Journal of Occupational and Organizational Psychology, 78, 387-409.

Borman, W. C., & Motowidlo, S. J. (1993). Expanding the criterion domain to include elements of

contextual performance. In N. Schmitt, & W. C. Borman, Personnel Selection in Organizations.

San Francisco: Jossey-Bass.

Borman, W. C., & Motowidlo, S. J. (1997). Task performance and contextual performance: The

meaning for personnel selection research. Human Performance, 10, 99-109.

Buhrmester, M., Kwang, T., & Gosling, S. D. (n.d.). Amazon's Mechanical Turk: A new source of

inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5.

Carlson, K. D., Scullen, S. E., Schmidt, F. L., Rothstein, H., & Erwin, F. (1999). Generalizable

biographical data validity can be achieved without multi-organziational development and

keying. Personnel Psychology, 52, 731-755.

Carroll, J. (1993). Human Cognitive Abilities: A Survey of Factor Analytic Studies. New York: Cambridge

University Press.

Cattell, R. B. (1941). Some theoretical issues in adult intelligence testing. Psychological Bulletin, 38, 592.

Cohen, J. (1992). A Power Primer. Psychological Bulletin, 112, 155-159.

Dye, D. A., Reck, M., & McDaniel, M. A. (2007). The validity of job measures. International Journal of

Selection and Assessment, 1, 153-157.


edX. (2015, February 25). Retrieved from discern: NY Times

Equal Employment Opportunity Commission, C. S. C. U. S. D. L. U., & Equal Employment

Opportunity Commission. (1978). Unifrom Guidelines on Employee Selection Procedures. Federal

register, 43(166), 38295-38309.

Fleisher, M. S., & Tsacoumis, S. (2012). O*NET analysit occupational skills ratings: Procedures update.

(Tech. Rep. No. FR-11-67) Alexandria, VA.: Human Resources Research Orgnaization

(HumRRO).

Fleisher, M. S., & Tsacoumis, S. (2012). O*NET analyst occupational abilities ratings: procedure update.

(Tech. Rep. No. FR-11-66). Alexandria, VA: Human Resources Research Organization

(humRRO).

Goldberg, L. R. (1992). The development of markers for the Big-Five factor structure. Psychological

Assessment, 4, 26-42.

Goldberg, L. R. (1993). The structure of phenotypic personlity traits. American Psychologist, 48, 26-34.

Hogan, J., & Holland, B. (2003). Using theory to evaluate personality and job-performance relations:

A socioanalytic perspective. Journal of Applied Psychology, 88, 100-112.

Horn, J. (1985). Handbook of Intelligence. New York: Wiley.

Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job

performance. Psychological Bulletin, 96, 72-98.

Hurtz, G. M., & Donovan, J. J. (2000). Personality and job performance: The Big Five revisited.

Journal of Applied Psychology, 85, 869-879.

Jaro, M. A. (1989). Advances in record linkage methodology as applied to the 1985 census of Tampa

Florida. Journal of the American Statistical ASsociation, 84, 414-420.

Jaro, M. A. (1995). Probalistic linkage of large public health data file. Statistics in Medicine, 14, 491-498.

Johnson, D. R., & Borden, L. A. (2012). Particiants at your fingertips: Using Amazon's Mechanical

Turk to increase student-faculyy collaborative research. Teaching of Psychology, 39, 245-251.

Judge, T. A., & Ilies, R. (2002). Relationship of personality to performance motivation: A meta-

analytic review. Journal of Applied Psychology, 87, 797-807.


Kanfer, R., Crosby, J. R., & Brandt, D. M. (1988). Investigating behavioral antecedenty of turnover

at three job tenure levels. Journal of Applied Psychology, 73, 331-335.

Kehoe, J. (2002). General mental ability and selection in private sector organizations: A commentary.

Human Performance, 15, 97-106.

Klimoski, R. J., & Childs, A. (1986). Successfully predicting career success: An application of the

biographical inventory. Journal of Applied Psychology, 71, 3-8.

Markoff, J. (2013, April 4). Essay-Grading Software Offers Professors a Break. NY Times. Retrieved

February 25, 2015, from http://www.nytimes.com/2013/04/05/science/new-test-for-

computers-grading-essays-at-college-level.html?_r=0

McCrae, R. R., & Costa, P. T. (1997). Personality trait structure as a human universal. American

Psychologist, 52, 509-516.

McCrae, R. R., & John, O. P. (1992). An introduction to the five-factor model and its applications.

Journal of Personality, 60, 175-215.

McCrae, R. R., Jang, K. L., Livesley, W. J., Riemann, R., & Angleitner, A. (2001). Sources of

structure: Genetic, environmental, and artifactual influences on the covariation of personality

traits. Journal of Personality, 69, 511-535.

McGrew, K. (2008). CHC theory and the human cognitive abilities project: Standing on the shoulers

of the giants of psychometric intelligence research. Intelligence, 37, 1-10.

Miller, J. D., Gentile, B., Wilson, L., & Campbell, W. K. (2013). Grandiose and vulnerable narcissism

and the DSM-5 Pathelogical Personality Trait Model. Journal of Personality Assessment, 95, 284-

290.

Minton, E., Gurel-Atay, E., Kahle, L., & Ring, K. (2013). Comparing data collection alternatives:

Amazon mTurk, college students, and secondary analysis. AMA Winter Educators' Conference

Proceedings, 24, (pp. 36-37).

Mount, M. K., & Barrick, M. R. (1995). The Big Five personality dimensions: Implications for

research and practice in human resources management. In K. R. (Eds.), Research in Personnel

and Human Resources Management (Vol. 13) (pp. 153-200). Greenwich, CT: JAI Press.

Navarro, G. (2001). A guided tour to approximate string matching. ACM Computing Surveys, 33, 31-

88.


Oswald, F. L., Schmitt, N., Kim, B. H., Ramsay, L. J., & Gillespie, M. A. (2004). Developing a

biodata measure and situational jusgment inventory as predictors of college student

performance. Journal of Applied Psychology, 89, 187-207.

Paruchuri, V. (2015, February 25). Retrieved from YouTube:

https://www.youtube.com/watch?v=zFeP678054U

Paunonen, S. V., Rothstein, M. G., & Jackson, D. (1999). Narrow reasoning about the use of broad

personaloty measures for personnel selection. Journal of Organizational Behavior, 20, 389-405.

Peterson, N. G., Mumford, M. D., Borman, W. C., Jeanneret, P. R., Fleishman, E. A., Levin, K.

Y., . . . Dye, D. M. (2001). Understanding work using the occupational information network

(O*NET): Implications for practice and research. Personnel Psychology, 54, 451-492.

Rothstein, H. R., Schmidt, F., Erwin, F. W., Owens, W. A., & Sparks, C. P. (1990). Biographical data

in employment selection: Can validities be made generalizable? Journal of Applied Psychology, 75,

175-184.

Saad, S., Carter, G. W., Rothenberg, M., & Israelson, E. (2014, March 9). Testing and Assessment: An

Employer's Guide to Good Practices. Retrieved from O*NET:

http://www.onetcenter.org/dl_files/empTestAsse.pdf

Salgado, J. E. (1997). The Five Factor Model of personality and job performance in the European

community. Journal of Applied Psychology, 82, 30-43.

Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., & De Fruyt, F. (2003). International validity

generalization of GMA and cognitive abilities: A European community meta-analysis.

Personnel Psychology, 56, 573-605.

Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., de Fruyt, F., & Rolland, J. P. (2003). A Meta-

Analytic Study of General Mental Ability Validity for Different Occupations in the

Eurpopean Community. Journal of Applied Psychology, 88, 1068-1081.

Salgado, J., & Anderson, N. (2003). Validity generalization of GMA tests across countries in the

European community. European Journal of Work & Organizational Psychology, 12, 1-17.

Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel

psychology: Practical and theoretical implications of 85 years of research findings.

Psychological Bulletin, 124, 262-274.


Schneider, R. J., Hough, L. M., & Dunnette, M. D. (1996). Broadsided by broad traits: How to sink

science in five dimensions or less. Journal of Organizational Behavior, 17, 639-655.

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass Corrlations: Uses in Assessing Rater Reliability.

Psychological Bulletin, 86, 420-428.

Society for Industrial Organizational Psychology. (2003). Principles for the Validation and Use of Personnel

Selection Procedures (4th ed.). Bowling Green, OH: Author.

Tett, R. P., & Burnett, D. D. (2003). A personality trait-based interactionist model of job

performance. Journal of Applied Psychology, 88, 500-517.

Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job

performance. Personnel Psychology, 44, 703-742.

U.S. Census Bureau. (2015, February 17). American Fact Finder. Retrieved from factfinder.census.gov:

factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?pid=ACS_12_1YR

_CP05&prodType=table

Wee, S., Newman, D. A., & Joseph, D. L. (2014). More than g: Selection quality and adverse impact

implications of considering second-stratum cognitive abilities. Journal of Applied Psychology, 99,

547-563.

Winkler, W. E. (1990). String comparator metrics and enhanced decision rules in the Fellegi-Sunter

Model of record linkage. Proceedsings of the Section on Survey Research Methods (pp. 354-359).

American Statistical Association.

Woo, S. E., Chernyshenko, O. S., Stark, S. E., & Conz, G. (2014). Validity of six openness facets in

predicting wok behaviors: A meta-analysis. Journal of Personality Assessment, 96, 76-86.


Appendix A: Summary of the HR Avatar Solutions

Solution O*NET SOC AIMS Form Behavioral

History Form Cognitive Work

Simulation Job Specific

Knowledge Test Essay Test

Typing Test

Account Manager 43-4051.00 Professional Professional General Office

Workplace Customer Service

Fundamentals √

Accountant / Auditor 13-2011.00 Professional Professional Business/Finance Accounting Concepts

√

Administrator - Elementary and

Secondary School 11-9032.00 Professional Professional


√

Agent - Purchasing 13-1023.00 Professional Professional Business/Finance

Aide - Home Health 31-1011.00 Professional Professional Face-to-Face

Customer Service

Aide - Personal Care 39-9021.00 Entry-Level Entry-Level Face-to-Face

Customer Service

Analyst - Financial 13-2051.00 Professional Professional Business/Finance Accounting Concepts

√

Analyst - Market Research

13-1161.00 Professional Professional Business/Finance Marketing Concepts √

Assembler - Electrical and Electronic

Equipment 51-2022.00 Entry-Level Entry-Level Basic Entry-Level

Electrician Fundamentals

Assembler / Fabricator - Other

51-2099.00 Entry-Level Entry-Level Basic Entry-Level

Attendant - Amusement /

Recreation 39-3091.00 Entry-Level Entry-Level Basic Entry-Level

Attendant - Food Services

35-3022.00 Entry-Level Entry-Level Face-to-Face

Customer Service

Attorney 23-1011.00 Professional Professional General Office

Workplace √

Baker 51-3011.00 Entry-Level Entry-Level Basic Entry-Level






Typing Test

Bank Teller 43-3071.00 Professional Professional Face-to-Face

Customer Service Banking

Fundamentals

Bartender 35-3011.00 Entry-Level Entry-Level Face-to-Face

Customer Service

Carpenter 47-2031.00 Entry-Level Entry-Level Basic Entry-Level Carpentry

Fundamentals

Cashier 41-2011.00 Professional Entry-Level Face-to-Face

Customer Service

Chief Executive 11-1011.00 Professional Professional Manager √

Childcare Worker 39-9011.00 Professional Professional Entry-Level Office Childcare

Fundamentals

Claims Adjuster, Examiner,

Investigator 13-1031.00 Professional Professional


√

Cleaner - Vehicles and Equipment


Clerk - Billing and Posting

43-3021.00 Entry-Level Entry-Level Entry-Level

Administration

Clerk - Bookkeeping, Accounting, and

Auditing 43-3031.00 Entry-Level Entry-Level


Accounting Concepts

Clerk - Counter / Rental


Customer Service

Clerk - File 43-4071.00 Entry-Level Entry-Level Entry-Level

Administration √

Clerk - General Office 43-9061.00 Entry-Level Entry-Level Entry-Level

Administration

Clerk - Hotel, Motel, Resort

43-4081.00 Professional Professional Face-to-Face

Customer Service Hospitality Concepts

Clerk - Information and Record Clerks


Administration √

Clerk - Insurance 43-9041.00 Professional Professional Entry-Level Business Insurance






Typing Test

Claims / Policy Processing

& Finance Fundamentals

Clerk - Order Processing


Administration √

Clerk - Payroll and Timekeeping


Administration √

Clerk - Production, Planning, and

Expediting 43-5061.00 Entry-Level Entry-Level


Clerk - Shipping / Receiving


Administration

Clerk - Stockroom 43-5081.00 Entry-Level Entry-Level Entry-Level

Administration

Coach / Scout 27-2022.00 Professional Professional General Office

Workplace √

Collections Specialist 43-3011.00 Professional Professional Remote Customer

Service

Compliance Officer 13-1041.00 Professional Professional General Office

Workplace √

Computer Applications Software

Developer 15-1132.00 Professional Professional


√

Computer Programmer

15-1131.00 Professional Professional Information Technology

√

Computer Programmer - Actionscript


Core ActionScript 3 √

Computer Programmer - C


Core C √

Computer Programmer - C++


Core C++ √

Computer 15-1131.00 Professional Professional Information Core Java √






Typing Test

Programmer - Java Technology

Computer Programmer - Java

EE 15-1131.00 Professional Professional


Java EE √

Computer Programmer -

Javascript 15-1131.00 Professional Professional


Core Java Script √

Computer Programmer - PHP


Core PHP √

Computer Programmer - Python


Python √

Computer Programmer - Ruby


Core Ruby √

Computer Programmer - Web

Developer 15-1131.00 Professional Professional


Html5/CSS3 √

Computer Programmer - Web

Developer with jQuery 15-1131.00 Professional Professional


Html5/CSS3 √

Computer Systems Analyst


√

Computer Systems Software Developer


√

Cook - Fast Food 35-2011.00 Entry-Level Entry-Level Basic Entry-Level Food Safety

Cook - Institution and Cafeteria

35-2012.00 Entry-Level Entry-Level Basic Entry-Level Food Safety

Cook - Restaurant 35-2014.00 Entry-Level Entry-Level Basic Entry-Level Food Safety

Cook - Short Order 35-2015.00 Entry-Level Entry-Level Basic Entry-Level Food Safety

Correctional Officer 33-3012.00 Professional Professional General Office

Workplace

Cost Estimator 13-1051.00 Professional Professional Entry-Level Business

& Finance






Typing Test

Counselor - Educational,

Guidance, School, Vocational

13-1051.00 Professional Professional General Office

Workplace √

Customer Service Face-to-Face

21-1012.00 Professional Entry-Level Face-to-Face

Customer Service

Customer Service Representative

43-4051.00 Professional Entry-Level Remote Customer

Service Customer Service

Fundamentals √

Data Entry Keyers 43-9021.00 Entry-Level Entry-Level Entry-Level

Administration √

Database Administrator


Relational Database Concepts

√

Dental Assistant 31-9091.00 Entry-Level Entry-Level Basic Entry-Level

Dental Hygienist 29-2021.00 Professional Professional Basic Entry-Level

Dishwasher 35-9021.00 Entry-Level Entry-Level Basic Entry-Level

Dispatcher - General 43-5032.00 Professional Professional General Office

Workplace

Driver - Heavy and Tractor-Trailer

53-3032.00 Professional Professional Entry-Level Office

Driver - Light Truck / Delivery


Workplace

Driver - Sales and Delivery

53-3031.00 Professional Professional Retail Sales

Driver - School Bus 53-3022.00 Entry-Level Entry-Level Basic Entry-Level

Driver Transit and Intercity Bus

53-3021.00 Professional Professional Basic Entry-Level

Electrical Engineer 17-2071.00 Professional Professional General Office

Workplace √

Electrician 47-2111.00 Entry-Level Entry-Level Technician Electrician

Fundamentals

Enforcement Officer 33-3051.00 Professional Professional General Office






Typing Test

Workplace

Engineer - Civil 17-2051.00 Professional Professional General Office

Workplace Construction Fundamentals

√

Engineer - Industrial 17-2112.00 Professional Professional General Office

Workplace √

Engineer - Mechanical


Workplace √

Executive Secretary / Administrative

Assistant 43-6011.00 Professional Professional Administration √ √

Fast Food Worker 35-3021.00 Entry-Level Entry-Level Face-to-Face

Customer Service

Firefighter 33-2011.00 Professional Professional Entry-Level Office

First Line Supervisor - Office and

Administrative Support

43-1011.00 Professional Professional First-Line Supervisor First Line Supervisor

Fundamentals √

First-Line Supervisor - Construction /

Extraction 47-1011.00 Professional Professional First-Line Supervisor

First Line Supervisor

Fundamentals & Construction Fundamentals

First-Line Supervisor - Food Preparation /

Serving 35-1012.00 Professional Professional First-Line Supervisor Food Safety

First-Line Supervisor - Helpers, Laborers,

and Material Movers 53-1021.00 Professional Professional First-Line Supervisor

First-Line Supervisor - Housekeeping and

Janitorial 37-1011.00 Professional Professional First-Line Supervisor

First-Line Supervisor - 49-1011.00 Professional Professional First-Line Supervisor First Line






Typing Test

Mechanics, Installers, Repairers

Supervisor Fundamentals

First-Line Supervisor - Non-Retail Sales


Fundamentals √

First-Line Supervisor - Production / Operations


Fundamentals √

First-Line Supervisor - Retail Sales


Fundamentals

First-Line Supervisor - Transportation and

Material-Moving 53-1031.00 Professional Professional First-Line Supervisor

Food Preparation Worker


Food Server - Nonrestaurant


General Project Manager

11-1021.00 Professional Professional Manager Project Management

Concepts √

Graphic Designer - Web Development


Workplace Html5/CSS3 &

HTML/CSS

Hairdresser, Hairstylist,

Cosmetologist 39-5012.00 Entry-Level Entry-Level Basic Entry-Level

Helper - Dining Room and Cafeteria


Helper - Production 51-9198.00 Entry-Level Entry-Level Basic Entry-Level

Host / Hostess - Restaurant


Customer Service

Fundamental Hospitality Concepts

Information 15-1199.09 Professional Professional Information Project Management √






Typing Test

Technology Project Manager

Technology Concepts

Inspector, Tester, Sorter, Sampler,

Weigher 51-9061.00 Professional Professional Technician

Installer / Repairer - Telecommunications

Equipment 49-2022.00 Entry-Level Entry-Level Basic Entry-Level

Janitor 37-2011.00 Entry-Level Entry-Level Basic Entry-Level

Laborer - Agricultural 45-2092.00 Entry-Level Entry-Level Basic Entry-Level

Laborer - Construction

47-2061.00 Entry-Level Entry-Level Basic Entry-Level Construction Fundamentals

Laborer - Freight and Warehouse


Laborer - Landscaping and Groundskeeping


Laborer - Packing / Packaging


Laundry and Dry-Cleaning Worker


Legal Secretary 43-6012.00 Professional Professional Administration √ √

Loan Interviewer 43-4131.00 Professional Professional Business/Finance Banking

Fundamentals √

Loan Officer 13-2072.00 Professional Professional Business/Finance Banking

Fundamentals √

Machinist 51-4041.00 Entry-Level Entry-Level Basic Entry-Level

Maid / Housekeeping Cleaner


Mail Carrier 43-5052.00 Entry-Level Entry-Level Face-to-Face

Customer Service






Typing Test

Management Analyst 13-1111.00 Professional Professional Business/Finance Accounting Concepts

√

Manager - Administrative

Services 11-3011.00 Professional Professional Manager √

Manager - Architectural and

Engineering 11-9041.00 Professional Professional Manager √

Manager - Computer and Information

Systems 11-3021.00 Professional Professional Manager √

Manager - Construction

11-9021.00 Professional Professional Manager Construction Fundamentals

Manager - Financial 11-3031.00 Professional Professional Manager Accounting Concepts

√

Manager - Food Service

11-9051.00 Professional Professional Manager Food Safety √

Manager - Human Resources

11-3121.00 Professional Professional Manager Human Resources

Fundamentals √

Manager - Industrial Production

11-3051.00 Professional Professional Manager √

Manager - Marketing 11-2021.00 Professional Professional Manager Marketing Concepts √

Manager - Medical and Health Services

(General) 11-9111.00 Professional Professional Manager

Health Care Administration (US)

√

Manager - Medical and Health Services

(US) 11-9111.00 Professional Professional Manager


√

Manager - Other 11-9199.00 Professional Professional Manager √

Manager - Real Estate and Community

Association 11-9141.00 Professional Professional Manager √






Typing Test

Manager - Sales 11-2022.00 Professional Professional Manager Sales Concepts √

Mechanic - Bus,Truck, Diesel

Engine 49-3031.00 Entry-Level Entry-Level Technician

Mechanic - Heating, Air Conditioning,

Refrigeration 49-9021.00 Professional Professional Technician

HVAC Fundamentals

Mechanic - Industrial Machinery

49-9041.00 Entry-Level Entry-Level Technician

Medical / Clinical Laboratory

Technologist 29-2011.00 Professional Professional Technician

Medical Assistant 31-9092.00 Professional Professional General Office

Workplace

Network and Computer Systems

Administrator 15-1142.00 Professional Professional


Computer Network Concepts

√

Nurse - Licensed Practical / Vocational


Workplace

Nurse - Registered 29-1141.00 Professional Professional General Office

Workplace √

Nursing Assistant 31-1014.00 Professional Professional Face-to-Face

Customer Service

Operating Engineer 47-2073.00 Professional Professional General Office

Workplace

Operator - Packaging / Filling Machines


Operator - Industrial Trucks / Tractors


Operator - Machine - Metal and Plastic


Painter - Construction 47-2141.00 Entry-Level Entry-Level Basic Entry-Level






Typing Test

and Maintenance

Paralegal /Legal Assistant


Workplace √ √

Personal Financial Advisor

13-2052.00 Professional Professional Business/Finance Accounting Concepts

√

Pharmacist (General) 29-1051.00 Professional Professional General Office

Workplace Health Care

Administration (US) √

Pharmacist (US) 29-1051.00 Professional Professional General Office

Workplace Health Care


Physical Therapist 29-1123.00 Professional Professional Entry-Level Office

Plumber, Pipefitter, Steamfitter

47-2152.00 Entry-Level Entry-Level Basic Entry-Level Plumbing

Fundamentals

Printing Press Operators


Production Worker 51-9199.00 Entry-Level Entry-Level Entry-Level Office

Radiologic Technologist


Workplace

Receptionist 43-4171.00 Entry-Level Entry-Level Entry-Level

Administration

Recreation Worker 39-9032.00 Entry-Level Entry-Level Basic Entry-Level

Recruiter 43-4111.00 Professional Professional General Office

Workplace Recruiting Concepts √

Retail Salesperson 41-2031.00 Professional Entry-Level Retail Sales Sales Concepts

Sales Agent - Insurance

41-3021.00 Professional Professional Business Sales

Sales Situation Analysis & Insurance

Fundamentals

Sales Agent - Real Estate

41-9022.00 Professional Professional Business Sales Sales Situation

Analysis & Sales Concepts

√

Sales Agent - 41-3031.00 Professional Professional Business Sales Sales Situation √






Typing Test

Securities, Financial Services


Sales Representative - Services

41-3099.00 Professional Professional Business Sales Sales Situation


√

Sales Representative - Technical and

Scientific 41-4011.00 Professional Professional Business Sales

Sales Situation Analysis &

Sales Concepts √

Sales Representative - Wholesale &

Manufacturing 41-4012.00 Professional Professional Business Sales

Sales Situation Analysis &

Sales Concepts √

Salesperson - Parts and Accessories

41-2022.00 Professional Professional Retail Sales

Secretary - Medical (General)

43-6013.00 Professional Professional Administration √

Secretary - Medical (US)

43-6013.00 Professional Professional Administration Health Care


Secretary / Administrative

Assistant 43-6014.00 Professional Entry-Level Administration √ √

Security Guard 33-9032.00 Professional Professional Entry-Level Office

Social / Human Service Assistant

21-1093.00 Professional Professional Entry-Level Office Social Services Fundamentals

Social Worker - Child, Family, School


Customer Service Social Services Fundamentals

Specialist - Computer Network Support


Computer Network Concepts

Specialist - Computer User Support


Customer Service Fundamentals

Specialist - Human Resources

13-1071.00 Professional Professional Manager Human Resources

Fundamentals √

Specialist - Office and 43-9199.00 Entry-Level Entry-Level Entry-Level √






Typing Test

Administrative Support

Administration

Specialist - Public Relations


Workplace Marketing Concepts √

Specialist - Training and Development


Workplace √

Taxi Driver / Chauffeur


Teacher - Elementary School


Workplace Education Delivery

Fundamentals √

Teacher - Health Specialties -

Postsecondary 25-1071.00 Professional Professional


Education Delivery Fundamentals

√

Teacher - Kindergarten



Fundamentals √

Teacher - Middle School



Fundamentals √

Teacher - Other 25-3099.00 Professional Professional General Office


Fundamentals

Teacher - Postsecondary



Fundamentals √

Teacher - Preschool 25-2011.00 Entry-Level Entry-Level Face-to-Face

Customer Service Education Delivery

Fundamentals

Teacher - Secondary School



Fundamentals √

Teacher - Self-Enrichment Education



Fundamentals √

Teacher - Special Education



Fundamentals √

Teacher - Substitute 25-3098.00 Professional Professional General Office


Fundamentals √






Typing Test

Teacher Assistant 25-9041.00 Professional Professional Entry-Level Office Education Delivery

Fundamentals

Team Assembler 51-2092.00 Entry-Level Entry-Level Basic Entry-Level

Technician - Automotive Service


Technician - Emergency Medical /

Paramedic 29-2041.00 Professional Professional Technician

Technician - General Maintenance and

Repair 49-9071.00 Professional Professional Technician

Technician - Medical and Clinical Laboratory

29-2012.00 Professional Professional Technician

Technician - Medical Records and Health

Information 29-2071.00 Professional Professional Entry-Level Office


Technician - Pharmacy

29-2052.00 Professional Professional Technician

Telemarketer 41-9041.00 Professional Professional Business Sales Sales Situation


Trainer - Athletic 39-9031.00 Professional Professional Entry-Level Office

Trimmer - Meat, Poultry, and Fish


Waiter / Waitress 35-3031.00 Entry-Level Entry-Level Face-to-Face

Customer Service

Welder, Cutter, Solderer, Brazer




Appendix B: Historical Validity Evidence for the

Cognitive Scales

Table 31. Compiled Validity Evidence for the Original Content of the Cognitive Work Simulation Scales

Performance Factor (various organizations)

Attention to Detail

Analytical Thinking

Listening Score (Organization A) .37

Performance Rating .50

Listening Score (Organization B) -.43

Listening Score (Organization C) -.35 -.34

Performance Rating (Organization A) .42

Performance Rating (Organization B) .39

Sales/Hour -.26 -.34

Cross Selling .37 .39

Response Quality .33

Average 2nd Contact .21

Schedule Conformance -.09

Performance Rating (Organization C) .39 .16

*Study results provided by original content developer. Sample sizes are all larger than 200 and p<.05


Appendix C: Historical Validity Evidence for AIMS

Tables 32-38 contain the results of criterion-related validity studies conducted by the developer of the original AIMS content. Please note that

these results are based on the longer, 100-item version of the assessment. Table 36 summarizes the statistically significant relationships (p<.05)

between the original AIMs scales and various measures of performance. The table includes results for over 5000 applicants and over 12 different

organizations from multiple industries including Financial Services, Insurance, Hospitality, Market Research, and Pharmaceuticals. Note that all

significant correlations are reported and although some of these correlations are negative, we would not expect all of the scales to be positively

related to all job performance dimensions (see above section on previous personality research).

Table 32. Study 1 Results: Concurrent Validation Study for Insurance Consultants N=122-136

Competency Performance

Appraisal Average

Policy Count Average

QRF Level Average

Idle Time Average CPH

Service Average Second

Contact

Needs Structure .18 -.15 .18

Innovative & Creative .16 .19

Enjoys Problem-Solving .24

Seeks Perfection

Exhibits a Positive Work Attitude -.19


Table 33. Study 2 Results: Concurrent Validation Study for Inside Sales N=105

Competency Perfectionism Quality

Innovative & Creative .26


Competitive

Seeks Perfection .19

Develops Relationships .28 .16

Expressive & Outgoing .27 .18

Corporate Citizenship -.17


-.18

Adaptable -.17

Table 34. Study 3 Results: Concurrent Validation Study for an Internet Services Order Processing N=84

Competency Quality


Develops Relationships -.27

Competitive -.22

Table 35. Study 4 Results: Concurrent Validation Study for an Internet and Cable Sales and Service Position N=72-93

Competency Performance Quality Attendance

Needs Structure .30 .26


Competitive

Seeks Perfection .21

Develops Relationships .24

Exhibits a Positive Work Attitude .25

Adaptable -.20

Table 36. Study 5 Results: Concurrent Validation Study Auto Rental Sales Role N=80-92

Competency Quality Yield Productivity

Needs Structure .30 .25 .29

Innovative & Creative .32

Seeks Perfection .34 .23

Adaptable .19


Table 37. Study 6: Concurrent Validation Study for Paramedics N=85

Competency Reading Learning

Office Procedures

Assessing Following Procedures

Dealing with

Difficulty

Treating Others

with Respect

On time

Overtime Flexibility Sick

Needs Structure

-.21 -.25

Innovative & Creative

-.26 -.15


-.27

.18 -.26 -.20

Competitive -.22 .20 .24

Seeks Perfection

-.21 .18


.19 -.21

Expressive & Outgoing

-.24 -.29


.24 .23


-.24 -.22

Adaptable .36 .26


Table 38. Summary of Relationships between Performance Measures and Original AIMS Scales

Performance Measure

Enjoys Proble

m Solving

Innovative and

Creative

Needs Structur

e

Adaptable

Develops Relationship

s

Expressive and

Outgoing

Competitive

Seeks Perfectio

n

Positive Work Attitud

e

Corporate Citizenshi

p

Total Score -.15 .20

Conversion Rate

-.15 -.16 .21 -.12

Calls Per Hour -.28 .22 -.17 -.17

Unavailability .18 .34 .14

Positive Attitude

Team Attitude -.14 .13 .14

Service Attitude

Caring Attitude .12

Ownership -.12

Performance -.13 .13

Range .13

Ranking -.12 .23 -.17

Sales -.13 -.15 -.10

Talk Time -.20

Account Weight -.14 -.18 -.14 -.14

Total Score -.49 .58

Conversion Rate

.47

Ranking (A) .20 .19

Ranking (B) .20 .28 .21 .21

Call Management

.26 .28 .27 .26 .19 .15 -.13

Average Hold Time

.15

Weighted Rating

.27

Supervisor Rating of Overall Performance

.23 .15


Performance Measure

Enjoys Proble

m Solving

Innovative and

Creative

Needs Structur

e

Adaptable

Develops Relationship

s

Expressive and

Outgoing

Competitive

Seeks Perfectio

n

Positive Work Attitud

e

Corporate Citizenshi

p

Supervisor Rating of Skill Acquisition

.34 .29 .25

Supervisor Rating of Summary Performance

-.41 .39

Supervisor Rating of Overall Performance

-.36 .28 .30


Appendix D: Scoring Rubric for Essays

Grammar Content Structure

Syntax, vocabulary, spelling, sentence structure, natural-sounding, command of English

Conciseness, appropriateness, addressed prompt

Logic, flow, organization, format

0 - 3 Latent with mechanical errors. May include errors in spelling, punctuation use, contractions, prepositions, missing words, verb conjugation and tense, etc. Contains a generally poor command of English. Errors are to the point that the writing is barely understandable, if at all.

Does not address more than half of requirements set up by prompt. Response may be incomplete or extremely off-topic. Response may have been written in a completely inappropriate tone for the intended audience. May be extremely unclear.

Ideas are so jumbled by illogical organization that they may be hard or impossible to follow. Format is not apparent or completely inappropriate for intended audience. There may be little to no transition between different ideas.

4 - 6 There are a few errors in grammar, but they do not severely hinder comprehension of the writing. English/wording may sound somewhat unnatural. Main idea of writing is still understood.

Addresses many or all requirements set up by prompt, but they may not have been thoroughly developed, may be missing information, or may be unclear. Response may be slightly off-topic.

Organization of ideas is slightly off-putting and confusing, but reader should be able to follow them and the main idea is still communicated. Ideas may lack transition when needed.

7 - 10 There may be one or two small errors, but they are very minor and do not affect comprehension of the writing. English sounds natural and flows well.

All requirements set up by the prompt are addressed clearly and well developed. There are no confusing spots. Response is not off-topic or missing information.

Organization of ideas is logical and easy to follow. Transitions are usually or always apparent where appropriate.


Appendix E: Directions for Rating Essay Tests

Scoring with a Rubric:

1. Read the rubric.

2. Read the prompt

3. Outline all requirements listed in the prompt

4. Read the response

5. Choose an appropriate score for the writing sample in each of the areas listed on the

rubric. Be sure to rate each area separately, and not to allow a good/bad score in one area to

affect the way you score another area in the same writing sample; a writing sample with

many spelling errors may still reflect all the required content.

Example of a writing sample with a perfect score:

Prompt:

Pretend you are an administrative assistant. Your boss wants to have an offsite team meeting to set

goals for you and the rest of the team next year. Write an email asking team members to attend an

all-day meeting on the first Monday of next month. Tell team members to write down their goals for

the year and come prepared to present them to the group. Additionally, ask if anyone wants to

volunteer to plan a team activity.


Response:

This person scored a 30/30 on their writing sample, on a basis of three factors: grammar, content,

and structure.

There are no errors in grammar or spelling. The English sounds natural and the writing flows well.

The writer scored a 10/10 for grammar.

The writer answered the question thoughtfully and thoroughly. Despite the fact that the author was

not presented with a date, time, or place in the prompt, he/she recognized that, in an actual work

setting, these figures would be required in a successful email, and included all necessary information.

The response addresses all requirements outlined in the prompt and leaves nothing unclear. 10/10

content.

The sample is structured logically so that it flows without abrupt jumps or changes in idea. It is easy

to read and follow. The writer also included a subject line and list of recipients, an appropriate

introduction (Team Members:) and conclusion (Thank you,). The sample received a 10/10 in

structure, earning it a 30/30 overall.

Example of a poor response to the same prompt:

Response:

We are going have a meeting on the first Monday next month. I want everybody to think of

some goals and write them to present to everyone. Does anyone want to plan a team activity?

Team Members:

Arthur is hosting a mandatory team meeting to set goals for next year. The meeting will take

place January 30, 2014 from 1:00 P.M. to 4:00 P.M. at the Ritz Carlton in Vienna, VA in

Conference Room B.

Arthur is expecting each of you to attend. Also, he is expecting you to prepare for the meeting

by documenting your goals and being prepared to present them to the team.

Arthur is looking for a volunteers to lead a team activity during the meeting. If you would like to

volunteer, contact Arthur with your idea by January 15th.

Thank you,


This person scored a 14/30 on their writing sample based on the same three factors: grammar,

content, and structure.

The grammar is understandable and there are no glaring errors, however, there is a missing word

(We are going have) and the wording is not always clear. The writer scored a 7 for grammar.

Only a few of the requirements set up by the prompt are addressed here, and when they are, the

ideas are hardly developed or elaborated on. Were this a real email, much would be left to confusion.

This person received a 3/10 for content.

There is little transition between ideas or logical flow in this response. Also, this response is not

structured in an email format. The sample received a 4/10 for structure, giving it a 14/30 overall.


Appendix F: Validity Evidence for HR Avatar Tests

The HR Avatar Automated Essay Scoring System

January, 2016

Introduction

Written communication is a key skill in many positions. Communicating via email, writing

reports, and creating presentations all require the ability to communicate effectively.

Traditionally, essays written for assessment purposes are scored using human raters and a pre-

defined scoring rubric. However, the cost of human scorers is relatively high and humans can

become fatigued and erratic in high volume situations.

Luckily, machine learning has advanced to the point where computers can substitute for human

raters reliably. The HR Avatar Essay Test is an implementation of this technique, which results

in lower cost and faster scoring turn-around.

The HR Avatar Essay Test consists of several writing prompts. The writing prompts were

designed to be general enough to provide an opportunity for anyone to be able to write a short

essay. It is easy to add additional prompts for specific situations or for general use.

Applicants are asked to write a short essay with a minimum of 100 words and are given an

unlimited time to do so. The essays are scored using Discern, an open source, machine learning

program. Discern was designed by edX, a nonprofit organization founded by Harvard and the

Massachusetts Institute of Technology (MIT) (edX, 2015; Markoff, 2013). The system produces

a score that ranges from 0 to 100. A confidence estimate for the score is also computed, which

ranges from 0 to 1. Scores with confidence estimates less than .10 are not considered valid.

How it works

HR Avatar uses open source essay scoring software originally published by EDX Corporation, a

spin-off of The Massachusetts Institute of Technology (MIT).

Software addresses and performs regression to produce a score for each submitted essay along a

continuous scale. This is different from classification, in which the software would simply

attempt to categorize each essay into one or more 'groups' or to rank the essays relative to one

another.

Each essay is written according to a predetermined set of instructions typically referred to as the

"Prompt." A typical prompt might be: "In a short essay of 100-400 words, explain whether it's

better to be a planner or to be a dreamer."

All essays are scored by the machine learning algorithm based on a "Training Set" upon which a

regression model has been built. The algorithm essentially analyzes all of the training essays and

produces a best guess at how the human scorers who created the training set would have judged

the new essay.


The application is written in Python and utilizes several open source machine-learning tools and

is centered around a machine-learning library called scikit-learn (http://scikit-learn.org), which in

turn uses a number of other open source mathematical and data manipulation packages

In order perform its task, the application converts each essay into a number of different

"features." Features are measurable aspects of the essay, such as spelling errors per character, or

grammar errors per character. In concept, the essay is reduced to an N-dimensional vector

containing all of the essay's feature scores. However, some features are complex vectors in and

of themselves.

Each feature is measured using specialized software for text analysis. For example, the grammar

errors are determined by looking for good and bad 'ngrams' which are essentially models of

either good or bad grammar.

Another important feature known as a "bag of words" is also generated. The bag-of-words model

is a simplifying representation used in natural language processing and information retrieval

(IR). In this model, a text (such as a sentence or a document) is represented as the bag (multiset)

of its words, disregarding grammar and even word order but keeping multiplicity. This results in

a vector with a length equal to the number of unique words in the largest essay evaluated.

It's helpful to understand the Bag of Words approach in terms of how it's used to filter out junk

email.

In Bayesian spam filtering, used by many spam filters, an e-mail message is modeled as an

unordered collection of words selected from one of two probability distributions: one

representing spam and one representing legitimate e-mail ("ham"). Imagine that there are two

literal bags full of words. One bag is filled with words found in spam messages, and the other

bag is filled with words found in legitimate e-mail. While any given word is likely to be found

somewhere in both bags, the "spam" bag will contain spam-related words such as "stock",

"Viagra", and "buy" much more frequently, while the "ham" bag will contain more words related

to the user's friends or workplace.

To classify an e-mail message, the Bayesian spam filter assumes that the message is a pile of

words that has been poured out randomly from one of the two bags, and uses Bayesian

probability to determine which bag it is more likely to be.

Along with the bag of words feature, another feature is generated that represents how 'topical' the

essay is, by using the bag of words vector that was generated.

Once the features are generated, the application formulates a model, using all training essays,

and their accompanying human-generated scores. The model is essentially a catalog of all feature

measurements for all of the training essays, along with their scores. Once created, this model can

then be used to determine where in the score space a new, unscored essay lies, based on its

feature measurements. In addition to score values, error values, which indicate how consistent

the training essay set was, can be calculated. This can provide a confidence value for the final

score.

The software uses a technique called Gradient Boosting Regression to pinpoint the score within

the model for a given essay by comparing the features for the new essay against the feature sets


of the pre-scored or 'training' essays. This is a well-established machine learning technique. Data

theory shows that this technique yields excellent results for regression problems like essay

scoring.

How does it perform?

The best indication of how well the machine learning algorithm works is to measure how well it

predicts the score a human rater would have come up with for any given essay. To do this we can

evaluate the machine-human rater reliability.

Reliability is a critical aspect of any assessment. It describes whether the score is consistent, and

puts an upper limit on the validity of the assessment. The data were analyzed to ascertain the

reliability of the machine scores of the essays to represent the scoring of human essay raters.

One thousand, two hundred and fourteen (N=1,214) essays were scored using both human

scoring and machine scoring. The correlation between the ratings was .73, representing an inter-

rater reliability of .73, which indicates that the machine scoring reliably rates the essays similarly

to human raters. In the world of testing, a reliability value of 0.73 is generally considered more

than acceptable.

Therefore, the machine scoring of the HR Avatar essay test was demonstrated to be a reliable

method for scoring essays that is similar to human ratings, but significantly more efficient,

requiring little or no human time or effort to arrive at an assessment of a large number of

applicants’ writing skills.


Appendix G: Validity Evidence for HR Avatar Tests

Results for the HR Avatar High Potential Solution

February 23, 2016

Validation studies have been conducted to confirm that the HR Avatar High Potential Solution

predicts job performance. Two separate studies are reported below. The first compared performance

on the assessment to a dichotomous outcome measure. In the second study, scores on the

assessment were correlated with a 1-100 performance rating.

Study 1

Three companies were included in the first study of the ability of the HR Avatar assessment to

predict manager job performance. The three companies were Johnson & Johnson, Harte Hanks, and

Manulife. They were identified as leaders in the Philippines in terms of using best practices in

personnel management. Individually, their sample sizes were too small to conduct separate

validation studies, but together the number of participants was high enough to enable confidence in

the results.

Results are reported below. The managers received a job performance rating of either “high” or

“marginal.” The sample included 130 managers. It is likely that the performance measure attenuated

the correlation because it is only two levels, which limits the amount of variance, accuracy, and

consistency it can provide. However, it does provide a measure of performance. Based on the

literature review of the components in the HR Avatar Assessment, we expect validity to be very

strong, approximately .35 - .40 uncorrected, and in the .50 - .60 range when corrected for criterion

unreliability.

Table 1 presents the results of the correlation analysis. The overall score predicted job performance

significantly (r=.25; p<.01). Although we are pleased to see a statistically significant correlation that

approaches .30, we believe this is an underestimate of the actual validity, due to the two-level

performance measure. Among the subcomponents, Attention to Detail was particularly robust in

predicting performance (r=.22; p<.05), as was Adaptable (r=.21; p<.05).

The high/marginal performance rating probably has low reliability. Thus, if we use a low reliability

of .40 for the correction for criterion unreliability, the validity increases to .63. We also used a higher

estimate for criterion reliability of .60, which is typically used as an estimate of reliability for well-

developed multi-level performance ratings, as an estimate of the reliability of the high/marginal

performance rating. The result was a more conservative correction for attenuation that yielded a

corrected correlation is .42.


Further support of the assessment’s prediction of job performance was provided by T-Test analyses,

which indicated that the high-performing group had a significantly higher average overall score

(X=63.16) on the assessment (p<.01) than those who were in the low-to-average job performance

group (X= 54.31).

In addition to predicting test performance, various scores on the assessment were related to

Leadership Engagement and Leadership Aspiration, suggesting that the underlying constructs are

related to important attitudes for leadership potential. For example, Needs Structure, Develops

Relationships, and Corporate Citizenship were related to Leadership Engagement. Further, several

test component scores were related to Leadership Aspiration, including Innovative and Creative,

Enjoys Problem Solving, Develops Relationships, and Adaptable.

Table 1: Correlations between Test Components and Job Performance and Job Attitude

Measures

Test Score Performance (High/

Marginal)

Leadership Engagement

Leadership Aspiration

Overall Score .25** .00 .16

Writing -.06 -.15 .09

Analytical Thinking .17 -.07 -.07

Attention to Detail .22* .01 .09

Adaptable .21* .10 .39**

Develops Relationships .02 .35** .50**

Enjoys Problem Solving .16 .26** .51**

Expressive and Outgoing -.01 -.13 .02

Innovative and Creative .18* .30** .67**

Needs Structure .02 .35** .36**

Seeks Perfection .13 .24** .36**

Frontline Management Fundamentals

.03 .03 .16

Corporate Citizenship .04 .28** .28**

Exhibits A Positive Work Attitude -.04 .11 .06

Competitive -.01 -.07 .00

*=p<.05; **=p<.01. N=130.

Study 2

The second study was conducted for an organization called UCPB LEAP. The sample was 64

managers who completed the HR Avatar assessment, and their scores were compared to their

previous year’s performance appraisal. The sample was small, which necessitated using a

nonparametric form of correlation called Spearman’s Rho, which indicates the extent to which the

rank order on the test was similar to the rank order on the performance measure. The correlation


was r=.25, which was significant (p<.05). This supports the assertion that managers who scored

higher on the assessment achieved higher performance ratings.

Conclusion

Based on the two studies presented above, we can say with confidence that managers who score

higher on the assessment perform better on the job. The uncorrected correlation of .25 between

overall score and the high/marginal performance measure is significant. When corrected for

criterion unreliability using a conservative approach, the correlation becomes .42. T-Tests also

support the same conclusion. More research is needed to build on the initial, yet promising results

described above. We expect that the results will demonstrate larger effect sizes and more robust

prediction of job performance when we are able to obtain measures of job performance that have

more variance and subjects are not simply placed into “high” and “marginal” categories. When we

have the time to adjust the scoring and weighting of the overall scores and have better criterion

measures, we believe it will be closer to .35 or more uncorrected, which would then yield a corrected

validity of approximately .60.


Appendix H: Emotional Intelligence Test

HR Avatar Emotional Intelligence Test

August 31, 2016

Emotional intelligence (EI; Sometimes called EQ) is the capacity to be aware of and understand

one’s own emotions, as well as other people’s emotions, and to use that information to manage

one’s own reactions and respond effectively in social situations. The concept gained considerable

attention in the 1990s, with the publication of a book called Emotional Intelligence (Goleman, 1995).

Since then, it continues to be researched and used to help people interact with others more

effectively. Emotional Intelligence has been measured in different ways by different test developers.

Self awareness, empathy, and self control are at the core of EI.

Previous validity evidence

The ability of EI to predict performance has been established. A growing body of research

demonstrates that EI predicts job success, as well as other important outcomes on the job (Carmeli,

2003; Farh et al., 2012; O’Boyle at al., 2011; Semadar et al., 2006; Van Rooy & Viswesvaran, 2004;

Wong & Law, 2002). One study, published by Cote and Miners in 2006, showed that employees

with relatively low cognitive intelligence can achieve strong job performance by compensating for it

with high emotional intelligence under the right conditions.

Most jobs include some degree of frustration, and require employees to control their emotions.

Those with higher EI tend to have better social relations with people, including management,

coworkers, and customers. This leads to a wide range of positive outcomes. Having highly-tuned

skills in sensing how other people are feeling and being aware of how one’s own emotions are

impacting one’s thinking can be very valuable in managing conflict, dealing with complex social

situations, and solving problems in team settings. This can help avert escalations of conflict, and

enable the person to solve problems proactively.

For example, a sales associate may be interacting with a customer. The customer is not

communicating very clearly about what they want, but they are giving subtle cues about their

communication style and how they are feeling. Someone who has low emotional intelligence may

misread what the person wants and respond in a way that is frustrating to that customer.

Alternatively, someone high in emotional intelligence is more likely to accurately interpret the

customer’s needs and respond in a way that effectively influences the customer, resulting in stronger

sales performance (Huggins et al, 2016).


High Emotional Intelligence can also improve work success by helping to:

Use mental capacity on work tasks instead of getting carried away with emotional reactions

Diffuse situations where there is potential for conflict and other nonproductive behaviors

Anticipate others’ reactions and adjust approach to enable effective communications

Not offend others, not alienate customers or partners, not cause problems

Demonstrate better impulse control, avoiding distractions and staying focused

One recent meta-analysis found a correlation of p=.29 between EI and supervisor-rated job

performance (Joseph et al., 2015). The authors characterize EI as a measure that is a shorthand

version that contains components of multiple established measures. The table below presents the

relationships they found between some of those related measures and job performance. Cognitive

ability demonstrated the highest power of prediction of success on the job. EI provides a snapshot

view of a certain combination of skills that are important for success at work.

Example Correlations between Test Performance and Supervisor Ratings of Job

Performance*

Test Type Validity

Cognitive Ability .44

Emotional Intelligence .29

Conscientiousness .21

Emotional Stability .11

Extraversion .09

* Source is Joseph et al. (2015) meta-analysis

Emotional Intelligence can also impact other important outcomes, such as organizational citizenship.

Research has shown that employees who have higher EI tend to focus more on the welfare of their

organization, and put more effort into actions that help the organization to function effectively, even

though the activities are not directly required of them (Shrestha & Baniya, 2016). Similarly, higher EI

is related to adaptability for frontline service employees, which in turn is related to job outcomes,

such as job satisfaction and job performance (Sony & Nandakumar, 2016).

There is debate about Emotional Intelligence. Some have suggested that EI is not really a new

concept, and that some of the published research may overstate its prediction of job performance

(e.g., Joseph et al., 2015). Rather, it is simply a clever combination and repackaging of other things

we already knew predicted job performance, and still does not relate to job performance as well as

cognitive ability. These are fair criticisms. Nonetheless, emotional intelligence has become as

popular as it has because it makes sense, and, as noted above, a growing body of research does show

that it predicts success on the job, as well as other important outcomes.


Measuring Emotional Intelligence

In roles where employees must interact with other people, for example, Customer Service, Sales, and

Management, the higher their capacity to be in touch with their own and others’ emotions,

understand those emotions, and behave in ways that are socially appropriate and demonstrate

impulse control, the better they will be received by other people. This pattern of skills and abilities

aligns with the observation that EI is related to Conscientiousness, Agreeableness, and Emotional

Stability, as well as cognitive ability. Emotional Intelligence can be used as one component of a good

selection process for hiring employees, as long as the potential overlap among different tools being

used is considered, and the combination of tools covers important skills for success on the job.

There are a number of available EI measures. Some focus on EI as an ability, and some are self-

report measures that represent EI in what is called a “mixed model,” which generally includes self

awareness, self regulation, social skill, empathy, and motivation. Three competencies in particular are

core to the concept of EI, because they involve the process of dealing with emotions directly. Being

aware of one’s own emotions and how one should behave in social situations, being aware of other

people’s emotions and caring about others, and being adept at impulse control and maintaining calm

can be thought of as “minimum requirements” for EI. Though its effectiveness probably is

overstated in certain publications, EI is a tool that can provide useful information about a candidate

in a relatively efficient way. A streamlined EI measure is described below.

Three Core Competencies of EI

EI Competency Definition Relation to Job Performance

Emotional Self Awareness

Monitors and understands how and why one reacts in particular ways to different situations, and knows how to conduct oneself appropriately and effectively in social situations

Interacts with customers and coworkers in an appropriate and measured way that reflects calm competence and inspires confidence

Empathy Senses and understands other people’s feelings, feels sympathy for other people, and sees things from other people’s point of view

Improves customer loyalty and wallet share through improved relationships, reducing levels of conflict in the workplace

Emotional Self Control Manages the desire to satisfy urges or impulses, shows restraint and manages behaviors to ensure appropriate and effective work habits and interactions with others, maintains composure in stressful situations

Prioritizes work tasks effectively, meeting long-term goals, and ability to form and leverage cooperative work relationships for better outcomes

The HR Avatar EI Test measures the three core components described above. They are three of the

most common aspects of the somewhat diverse set of competencies that make up EI. HR Avatar’s

EI Test is a streamlined mixed-model job-related questionnaire, and is offered as a new addition to


HR Avatar’s personality assessment to provide an alternate window into candidates’ interpersonal

skills, without having to use a separate test. It measures the core competencies of EI in a self-report

assessment.

An example EI item: I can usually tell how other people are feeling.

It’s important to note that while EI can help predict success on the job, it provides just one piece of

the puzzle for predicting success on the job. The best way to evaluate a candidate is to measure

multiple traits, including cognitive ability, past behavior and related job knowledge.

Reliability

The descriptive statistics and reliability estimates of the EI scales were estimated using data collected

via MTurk. As can be seen from the table below, the EI scales have acceptable levels of internal

consistency.

Descriptive Statistics and Reliability Evidence for the EI Scales

Competency N M SD Alpha

Emotional Self Awareness 320 25.84 4.25 .67

Empathy 320 25.38 4.34 .70

Emotional Self Control 320 25.18 5.49 .78


References

Carmeli, A. (2003). The relationship between emotional intelligence and work attitudes, behavior

and outcomes: An examination among senior managers. Journal of Managerial Psychology, 18(8),

788-813.

Cote, S., Miners, C.T.H. (2006). Emotional intelligence, cognitive intelligence, and job performance.

Administrative Science Quarterly, 51, 1-28.

Farh, C.I.C.C., Seo, M., Teslluk, P.E. (2012). Emotional intelligence, teamwork effectiveness, and

job performance: The moderating role of job context. Journal of Applied Psychology, 97(4), 890-900.

Goleman, D. (1995). Emotional Intelligence, New York, NY, England: Bantam Books, Inc.

Huggins, K.A., White, D.W., & Stahl, J. (2016). Antecedents to sales force job motivation and

performance: The critical role of emotional intelligence and affect-based trust in retailing managers.

International Journal of Sales, Retailing & Marketing, 5(1), 27-37.

Joseph, D.L., Jin, J., Newman, D.A., & O’Boyle, E.H. (2015). Why does self-reported emotional

intelligence predict job performance? A meta-analytic investigation of mixed EI. Journal of Applied

Psychology, 100(2), 298-342.

O’Boyle, E.H., Humphrey, R.H., Pollack, J.M., Hawver, T.H., & Story, P.A. (2011). The relation

between emotional intelligence and job performance: A meta-analysis. Journal of Organizational

Behavior, 32, 788-818.

Semadar, A., Robbins, G., & Ferris, G.R. (2006). Comparing the validity of multiple social

effectiveness constructs in the prediction of managerial job performance. Journal of Organizational

Behavior, 27(4), 443-461.

Shresna, A.K., Baniha, R. (2016). Emotional intelligence and employee outcomes: Moderating role

of organizational politics. Business Perspectives & Research, 4(1), 15-26.

Sony, M., & Nandakumar, M. (2016). The relationship between emotional intelligence, frontline

employee adaptability, job satisfaction and job performance. Journal of Retailing & Consumer

Services, 30, 20-32.

Van Rooy, D.L., Viswesvaran, C. (2004). Emotional intelligence: A meta-analytic investigation of

predictive validity and nomological net. Journal of Vocational Behavior, 65(1), 71-95.


Wong, C.S., Law, K.S. (2002). The effects of leader and follower emotional intelligence on

performance and attitude: An exploratory study. The Leadership Quarterly, 13(3), 243-274.


Appendix I: Work Competency Tests

HR Avatar Work Competency Tests

August 31, 2016

The HR Avatar Work Competency Tests were developed to address the need for assessments of

workplace competencies that focus on interpersonal skills. The Work Competency Tests use a

situational judgment test (SJT) format, in which candidates make decisions in situations that

employees typically encounter at work. The SJT format has proven particularly effective at

measuring critical workplace competencies, which can impact the overall performance and bottom

line of an organization.

SJT’s have been used for many years to predict job performance (Christian, Edwards & Bradley,

2010; McDaniel et al, 2007). Christian et al (2010) conducted a meta-analysis that considered the

ability of SJT’s to predict job performance in a number of domains. SJT’s that measure applied

social skills predicted job performance ranging from r=.25 to r=.38 (See below Table). Additionally,

Christian et al (2010) found that video-based SJT’s tend to predict performance better than paper-

and-pencil tests.

Results of SJTs that Measure Applied Social Skills in Predicting Job Performance*

Domain Number of studies Number of people Validity**

Interpersonal skills 17 8,625 .25

Teamwork skills 6 573 .38

Leadership 51 7,589 .28 *From Christian et al (2010) meta-analysis.

**Validity corrected for criterion unreliability.

Content Development

Four assessments were created, one for each of the following job families: 1) Team Member, 2)

Service, 3) Sales, and 4) Leader. Content was developed by conducting research on job descriptions

and typical activities conducted within each job family. Competency models were created for each

job family, and a panel of SMEs was engaged to develop scenarios for each test. Responses

represented different course of action that could be taken, given the scenario. Test-takers are asked

to rate the effectiveness of each. SMEs were further engaged to generate alternative actions for each

scenario, and to estimate the effectiveness of each response.

MTurk was used to gain additional SME evidence for building the assessment. SMEs from each job

family completed the assessments, and provided their ratings of the effectiveness of each alternative

(Team Member N=49, Service N=49, Sales N=52, Leader N=49). Their mean responses were used


as the true score for each of the assessments. The test-taker score is the difference between

responses and the true scores, combined for each competency. SMEs also rated the extent to which

the scenarios appeared to be similar to things that happen in real work situations relevant to the job

family, and whether each competency was important for performing the job scenario. Ratings for all

competencies for all four job families supported the assertion that the scenarios were similar to real-

life job situations and the competencies were important for performing the situation represented in

the scenario.

In the HR Avatar Work Competency Tests, the work competency scenarios are presented using HR

Avatar’s simulation technology. First, a general description of the context is presented. Then, the

test-taker views a scene in which characters are interacting, and a situation occurs. They then

respond to a list of 5-8 alternative responses, indicating how effective each response is on a scale of

1-5.

Sample Work Competency Test Item

Context: You work on a team that has an important deadline coming up.

Coworker: Female: You know how we are all supposed to get our work done by tomorrow close of business? Well I just overheard Jim telling Jessica that he is going to a concert tonight and will probably call in sick tomorrow. He also said that he’s not even close to getting his assignment finished, so it doesn’t matter anyway. What should I do?

Rate the effectiveness of the following responses:

1. Tell Jim you are going to tell the Team Leader what you heard if he calls in sick tomorrow. 2. Talk to the team leader now to share what you heard. 3. Just ignore it and let Jim fail on his own. 4. Put an anonymous note on the Team Leader’s desk explaining what you overheard.

Team Member Work Competency Test Description

The Team Member Work Competency Test was designed to assess critical competencies within jobs

where the employee must interact with others, both inside and outside an organization, in order to

be successful. Virtually all jobs require interaction, so this assessment is a good choice for just about

any job.

Team Member Work Competencies

Building Relationships with Customers and Coworkers: Effective work relationships

help produce better results, because most roles in organizations require interacting with

others and relying on them to accomplish objectives. This includes developing constructive


and cooperative working relationships with others, and maintaining them over time. It also

involves effectively managing 1:1 interactions with others, maintaining positive relationships

while accomplishing objectives, and building credibility through mutual trust and respect.

Demonstrating Customer Focus: Customers are the reason most organizations exist.

Keeping the customer in mind and providing a positive customer experience is critical to

success. Demonstrating Customer Focus includes working with customers to understand

needs and ensure that products and services meet their needs, doing what can be done to

resolve issues, getting the customer to someone who can help if needed, and following up to

ensure there is resolution. It also involves handling the situation in a conscientious and

customer-focused way.

Resolving Conflicts and Meeting Customer Needs: Being able to effectively resolve

conflicts and meet customer needs is a critical aspect of being an effective team member. It

involves creatively solving problems, maintaining a calm demeanor, and managing through

differences of opinion. It includes handling complaints, looking for ways to solve problems

collectively and agree on next steps, settling disputes and resolving grievances and conflicts,

or otherwise negotiating with others. Additionally, effective conflict management requires

the team member to work to understand the views of both sides of a disagreement, ensure

relevant information is shared and considered, and help the parties in a conflict to find

common objectives.

Working Well with Teams: When team members are cooperative, helpful, and respectful

of others, it results in a more positive and cooperative workplace. Demonstrating good

teamwork results in other people feeling good about working with you, and increases the

chances that they will be helpful in return. This involves working effectively with other

people and teams, supporting and showing respect for others, showing interest in other

people’s work, and saying positive things about the work and organization. It also involves

helping others get their work done, making sure the team’s work gets done, and being ready

to put team goals ahead of your own individual goals.

Maintaining Flexibility and Adaptability: Organizations face an increasing amount of

change, whether due to technology, mergers or acquisitions, or changing strategy to keep up

with the competition. Flexibility and adaptability can help you be effective in organizations

during times of change. This includes responding well to change, modifying your approach

in light of new demands, and adapting to, supporting, and accepting change. It also involves

demonstrating resilience, hardiness, and effective coping skills during difficult times.


Customer Service Work Competency Test Description

The Customer Service Work Competency Test was designed to assess critical competencies within

jobs where the employee is required to provide customer service in order to be successful.

Customer Service Work Competencies






















common objectives.

Working Well with Teams: When team members are cooperative, helpful, and respectful

of others, it results in a more positive and cooperative workplace. Demonstrating good

teamwork results in other people feeling good about working with you, and increases the

chances that they will be helpful in return. This involves working effectively with other

people and teams, supporting and showing respect for others, showing interest in other

people’s work, and saying positive things about the work and organization. It also involves

helping others get their work done, making sure the team’s work gets done, and being ready

to put team goals ahead of your own individual goals.


Helping Customers Understand Information: Communicating effectively with customers

is at the core of delivering customer service. It depends largely on how effectively you help

customers understand information. Customers tend to have a wide range of knowledge

levels, so it is important to meet them where they are and provide the appropriate level of

detail to ensure their understanding, in a way that makes them feel good about the

experience. This involves asking questions to understand their needs, issues, and current

level of understanding, translating or explaining what information means and how it can be

used, and providing a sufficient amount of detail. It also includes demonstrating empathy

with the other person, helping the customer to review information, and looking for details

that will help the customer understand and/or address the problem.

Sales Work Competency Test Description

The Sales Work Competency Test was designed to assess critical competencies within jobs where

the employee is required to make sales in order to be successful.

Sales Work Competencies























common objectives.

Driving for Results: Being successful at sales generally requires a relentless persistence and

focus on goals. The constant pursuit of closing deals will translate into increased sales. This

requires you to push yourself to achieve objectives, stay focused on making sales,

aggressively close deals, and to work toward achieving both personal and team goals. It also

involves asking questions to determine what obstacles there are and to find the best way to

drive to making a sale, and influencing customers to buy.

Conveying Value: Conveying value to customers is critical to sales success. It requires

flexibility in messaging, thinking about what you have to offer, and understanding the

context of the customer and the sale. It requires some degree of creativity. Conveying value

involves recognizing and describing value to customers, determining customer needs and

connecting value to those needs, and influencing others. It also involves describing how

value is related to important customer outcomes (e.g., saving time, saving money, increasing

speed to market).

Leader Work Competency Test Description

The Leader Work Competency Test was designed to assess critical competencies within jobs where

the employee is required to manage a team in order to be successful.

Leader Work Competencies

Guiding, Directing, and Motivating Others: Guiding, Directing, and Motivating Others

is a core aspect of managing people. It is how a manager gets the team to act, and how a

manager ensures that action is effective. It involves providing direction and guidance to

subordinates, including setting performance standards and monitoring performance. It also

includes coordinating the work and activities of others, encouraging goal accomplishment,

making detailed plans that consider what is most important, and communicating priorities to

team members. To be effective at guiding others, a leader must effectively hold the team

accountable for their work, and provide advice that is reasonable and socially aware.

Coaching and Developing Others: Coaching and Developing Others has a significant

impact on a leader's long-term effectiveness. It involves understanding employees' needs and

helping them to grow and improve their job performance. It includes identifying the

development needs of others and coaching, mentoring, or otherwise helping others to

improve their knowledge or skills. Effective coaching and developing starts with building a

relationship of mutual trust, working together to decide what to accomplish, goal(s) to set,


and developing a roadmap for reaching the goal, and giving feedback along the way. The

leader should provide specific behavioral examples when giving feedback on performance

issues, clarify expectations, and get a commitment from the employee to act.


conflicts and meet customer needs is a critical aspect of effective leadership. It involves

creatively solving problems, maintaining a calm demeanor, and managing through




the leader to work to understand the views of both sides of a disagreement, ensure relevant

information is shared and considered, and help the parties in a conflict to find common

objectives.

Exercising Political Savvy: Building relationships and managing impressions with

decision-makers and others who have influence in organizations can help you be effective

and successful. When they are deciding whom to include in strategic discussions and where

to put resources, people are more likely to think to include those they know and trust.

Political savvy can help you to be more likely to be included in decisions made by leaders in

your organization. Political savvy includes understanding how to position yourself and

communicate objectives in the context of organizational issues and other personnel to

maximize outcomes both for your group and the organization, getting people to cooperate

with you, socializing your ideas, and building bridges to meet others halfway.

Team Building: Teams that are characterized by trust, support, and positive interactions

tend to be more effective. Leaders can help facilitate positive interactions among their teams.

Team building involves engaging and participating in activities that support improved team

social relations, building mutual trust, respect, communication, understanding and

cooperation among team members. It also includes focusing on providing a team

environment that is conducive to collaboration, fostering innovation and creativity,

promoting increased comfort level and celebration among team members.


References

Christian, M.S., Edwards, B.D., & Bradley, J.C. (2010). Situational judgment tests: Constructs

assessed and a meta-analysis of their criterion-related validities. Personnel Psychology, 63, 83-117.

McDaniel, M.A., Hartman, N.S., Whetzel, D.L, & Grubb III, W.L. (2007). Situational judgment tests,

response instructions, and validity: A meta-analysis. Personnel Psychology, 60, 63-91.

Documents

HR Avatar Assessment Solution Technical Manual · HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 1 HR Avatar Assessment Solution Technical Manual August 31, 2016