Whats Up DOT? IARP Occupational Database Committee IARP Forensic Conference Hyatt Regency Bonaventure Weston, Florida October 31, 2008

What’s Up What’s Up DOT?DOT?

IARP Occupational Database IARP Occupational Database CommitteeCommittee

IARP Forensic ConferenceIARP Forensic Conference

Hyatt Regency BonaventureHyatt Regency Bonaventure

Weston, FloridaWeston, Florida

October 31, 2008October 31, 2008

Scope of ProjectScope of Project

IODC → formed spring 2007IODC → formed spring 2007 Purpose: Purpose:

• Identify replacement of DOTIdentify replacement of DOT Focus: Focus:

• research past efforts to update DOTresearch past efforts to update DOT• current uses of DOT & O*NETcurrent uses of DOT & O*NET• identify & evaluate other identify & evaluate other

occupational databasesoccupational databases• develop support to update/replace develop support to update/replace

DOTDOT

Scope of ProjectScope of Project Identified two occupational databases: Identified two occupational databases:

eDOT, a product of Economic eDOT, a product of Economic Research Institute; McDOT, a product Research Institute; McDOT, a product of Vocationology, Inc. of Vocationology, Inc.

Third database, based on the Common Third database, based on the Common Metric Questionnaire (CMQ) may be Metric Questionnaire (CMQ) may be ready in few yearsready in few years

Arranged to have training and use of Arranged to have training and use of these databases for study purposes.these databases for study purposes.

First, needed to learn more about DOTFirst, needed to learn more about DOT

History of the DOTHistory of the DOT Following Civil War, U.S. and Census Following Civil War, U.S. and Census

Bureau began to classify occupationsBureau began to classify occupations First occupational dictionary: 1927, First occupational dictionary: 1927, A A

Dictionary of Occupational Terms,Dictionary of Occupational Terms, Great Britain Ministry of LabourGreat Britain Ministry of Labour

Wagner Peyser Act, 1933: United Wagner Peyser Act, 1933: United States Employment Service (USES)-- States Employment Service (USES)-- match workers with jobs.match workers with jobs.

History of the DOTHistory of the DOT

Work on the DOT began around the Work on the DOT began around the same time; 1same time; 1stst Edition 1939 Edition 1939

Subsequent editions published 1949, Subsequent editions published 1949, 1965, 19771965, 1977

Supplements came out 1982, ’86, ’91Supplements came out 1982, ’86, ’91 20 new occupations added in 199820 new occupations added in 1998 Intended purpose: to assist USES in Intended purpose: to assist USES in

placing workers in jobs.placing workers in jobs. 11stst Edition: skilled, semi-skilled or Edition: skilled, semi-skilled or

unskilledunskilled

History of the DOTHistory of the DOT 1965 edition: worker traits, worker 1965 edition: worker traits, worker

functions (Data, People, Things), functions (Data, People, Things), Occupational Group ArrangementsOccupational Group Arrangements

1977 edition: ~75,000 job analyses; 1977 edition: ~75,000 job analyses; 2,100+ new occupations; eliminated 2,100+ new occupations; eliminated gender bias in job titles and gender bias in job titles and descriptions; included detailed worker descriptions; included detailed worker characteristics (SCO)characteristics (SCO)

1986 ed. added 761 new occupations1986 ed. added 761 new occupations 1991 added 844, deleted 208; added 1991 added 844, deleted 208; added

GOE codes, GED, SVP, DLUGOE codes, GED, SVP, DLU

History of the DOTHistory of the DOT Each definition includes: code, title, Each definition includes: code, title,

industry designation, alternate titles, industry designation, alternate titles, body of definition, undefined related body of definition, undefined related titles, and definition trailer (GOE, titles, and definition trailer (GOE, strength, GED-RML, SVP, DLU)strength, GED-RML, SVP, DLU)

Basic concepts described in the DOT Basic concepts described in the DOT occupational definitions: element, occupational definitions: element, task, position, job, and occupationtask, position, job, and occupation..


Major elements of an occupation: Major elements of an occupation: • What the worker does (data, people, What the worker does (data, people,

things)things)• What gets done (work fields-WF)What gets done (work fields-WF)• Skill level to perform this work (SVP)Skill level to perform this work (SVP)• End product (materials, products, End product (materials, products,

subject matter and services-MPSMS)subject matter and services-MPSMS)


DOT → 4 classification systems: DOT → 4 classification systems: • by job content (OGA)by job content (OGA)• by worker function (DPT)by worker function (DPT)• by industry affiliation (Industry by industry affiliation (Industry

Designation)Designation)• alphabetically by title alphabetically by title

History of the DOTHistory of the DOT Job analysis → basis of the DOTJob analysis → basis of the DOT National Research Council (1999) National Research Council (1999)

defines occupational [job] analysis: defines occupational [job] analysis: "the tools and methods used to "the tools and methods used to describe and label work, positions, describe and label work, positions, jobs and occupations"jobs and occupations"

DOT defines an occupation as: “a DOT defines an occupation as: “a collective description of individual collective description of individual jobs performed, with minor variations, jobs performed, with minor variations, in many establishments”in many establishments”

History of the DOTHistory of the DOT The The Handbook for Analyzing JobsHandbook for Analyzing Jobs → →

job analysis methodology used in job analysis methodology used in creation of DOT.creation of DOT.

HAJ first published in 1944; 4HAJ first published in 1944; 4thth Edition Edition in 1991 (RHAJ).in 1991 (RHAJ).

1972 Edition published halfway 1972 Edition published halfway through completion of 1977 DOT.through completion of 1977 DOT.

Two primary formats in job analysis: Two primary formats in job analysis: work-oriented—what gets done; and work-oriented—what gets done; and worker-oriented—what the worker worker-oriented—what the worker does. HAJ/RHAJ uses both formatsdoes. HAJ/RHAJ uses both formats


The The Selected Characteristics of Selected Characteristics of Occupations Defined in the Revised Occupations Defined in the Revised Dictionary of Occupational TitlesDictionary of Occupational Titles (1993) (1993)

1st edition 1966, updates in 1968, 1st edition 1966, updates in 1968, 1981, and 1993.1981, and 1993.

Includes DOT titles arranged by: Includes DOT titles arranged by: GOE;GOE; SVP; strength; physical demands; SVP; strength; physical demands; environmental conditions; index of environmental conditions; index of titles with DOT codes; definitions of titles with DOT codes; definitions of the worker traits and functionsthe worker traits and functions


Positive factors of the DOT:Positive factors of the DOT: • use of skilled job analysts use of skilled job analysts • task level descriptionstask level descriptions• foundation built upon data people foundation built upon data people

thingsthings• Worker traits, characteristicsWorker traits, characteristics• Useful for TSA: MPSMS, WF, SVPUseful for TSA: MPSMS, WF, SVP• Attempt to cover national economyAttempt to cover national economy• Depth of informationDepth of information

Development of the Development of the DOTDOT

Sample for DOT: all jobs in the US Sample for DOT: all jobs in the US economyeconomy

County Business Patterns/Thomas County Business Patterns/Thomas Registry → used to identify # Registry → used to identify # establishments in each industry.establishments in each industry.

Industries assigned to one of 11 field Industries assigned to one of 11 field centers → average 42 industries/field centers → average 42 industries/field centercenter

““any industry” designation given to one any industry” designation given to one field officefield office


Each analyst identified which Each analyst identified which establishments to contact establishments to contact

Attempt to include one small, one medium Attempt to include one small, one medium and one large size establishment for each and one large size establishment for each industry, and to focus on new and industry, and to focus on new and emerging occupations. emerging occupations.

No clear supervision during process.No clear supervision during process. Appeared primary criteria for selecting Appeared primary criteria for selecting

establishment → proximity to field officeestablishment → proximity to field office Employers: right of refusal; no incentivesEmployers: right of refusal; no incentives


Employers controlled which jobs were Employers controlled which jobs were analyzed, and which employees were analyzed, and which employees were chosen for analysis.chosen for analysis.

Analyst chose which jobs to observe.Analyst chose which jobs to observe. If job analyzed for the 3If job analyzed for the 3rdrd Edition, may Edition, may

complete abbreviated analysis or none complete abbreviated analysis or none at all.at all.

If another analyst had recently If another analyst had recently completed a similar analysis, none completed a similar analysis, none would be done.would be done.

Observe 1-2 workers per job.Observe 1-2 workers per job.


Variety methods to record data: HAJ Variety methods to record data: HAJ format; abbreviated format; or simple format; abbreviated format; or simple notes.notes.

Analysts not allowed to bring in tools Analysts not allowed to bring in tools or equipment to measure certain or equipment to measure certain aspects of jobs—estimation and aspects of jobs—estimation and observation only. observation only.

Methods for rating worker traits were Methods for rating worker traits were vague—particularly for aptitudes, vague—particularly for aptitudes, temperaments and interests.temperaments and interests.

Updates of the DOT Updates of the DOT

Much time has passed since any onsite Much time has passed since any onsite job analyses of DOT occupationsjob analyses of DOT occupations

Date last Date last analyzed analyzed

NN %% AgeAge

1977 and 1977 and previousprevious

10,28810,288 80.680.6 30 years30 years

19781978 355355 2.82.8 30 years30 years

1979-19821979-1982 520520 4.14.1 >25 >25 yearsyears

1984-19881984-1988 1,1871,187 9.39.3 >20 >20 yearsyears

1989-19911989-1991 390390 3.13.1 >15 >15 yearsyears

19931993 2020 0.20.2 15 years15 years

How the DOT is usedHow the DOT is used

Basis for transferable skills analysisBasis for transferable skills analysis Primary uses: career & voc Primary uses: career & voc

counseling, SSDcounseling, SSD Secondary uses: library reference, Secondary uses: library reference,

personnel management, employee personnel management, employee placement, govt. uses, research, placement, govt. uses, research, othersothers

O*NETO*NET

Outgrowth of the Advisory Panel on the Outgrowth of the Advisory Panel on the DOT (APDOT) from the early 1990sDOT (APDOT) from the early 1990s

Dept. of Labor “replacement” for the Dept. of Labor “replacement” for the DOTDOT

Skills-based database rather than task-Skills-based database rather than task-based dictionarybased dictionary

Preliminary version released Dec. 1997 Preliminary version released Dec. 1997 with first version (O*NET 98) out Dec. with first version (O*NET 98) out Dec. 19981998

O*NETO*NET Utilized SOC coding rather than DOTUtilized SOC coding rather than DOT Current version is O*NET 13Current version is O*NET 13 The 12, 761 DOT occupations have been The 12, 761 DOT occupations have been

aggregated/collapsed to 812 groupsaggregated/collapsed to 812 groups Composite information from many jobs; Composite information from many jobs;

not intended to describe a particular jobnot intended to describe a particular job Uses mean data rather than mode used Uses mean data rather than mode used

by DOTby DOT

Position Analysis Position Analysis Questionnaire (PAQ)Questionnaire (PAQ)

PAQ’s business: create useful work and PAQ’s business: create useful work and labor economic reportslabor economic reports

Structured job analysis; 187 items (plus Structured job analysis; 187 items (plus 8 items re: comp.) called “job elements” 8 items re: comp.) called “job elements”

Job elements are worker-oriented;6 Job elements are worker-oriented;6 domains: information input; work domains: information input; work output (physical activities and tools); output (physical activities and tools); mental processes; relationships with mental processes; relationships with others; job context (physical and social others; job context (physical and social environment); and other work environment); and other work characteristics (such as pace and characteristics (such as pace and structure)structure)

PAQPAQ PAQ dedicated solely to job analysis PAQ dedicated solely to job analysis

services since 1972services since 1972 Terminology, definitions and rating Terminology, definitions and rating

scales different than that used in DOTscales different than that used in DOT Requires post-college graduate Requires post-college graduate

reading levelreading level Job analysts or supervisors usually Job analysts or supervisors usually

complete PAQcomplete PAQ

PAQPAQ

PAQ trains subscriber employers in PAQ trains subscriber employers in job analysis techniques—in person and job analysis techniques—in person and onlineonline

Limited coverage of managerial, Limited coverage of managerial, supervisory, executive and supervisory, executive and professional work led to development professional work led to development of separate questionnaireof separate questionnaire

Average reliabilities—item and re-rateAverage reliabilities—item and re-rate—range .68-.80—range .68-.80

PAQ/DAQPAQ/DAQ DAQ: Disability Analysis DAQ: Disability Analysis

Questionnaire. Developed by PAQ Questionnaire. Developed by PAQ Includes 99 questions from the PAQ Includes 99 questions from the PAQ

that are most closely related to items that are most closely related to items from the DOT/HAJfrom the DOT/HAJ

ERI modified DAQ slightly—made ERI modified DAQ slightly—made certain the questions best-matched certain the questions best-matched those of the DOTthose of the DOT

The DAQ questions are the ones asked The DAQ questions are the ones asked of incumbents when visiting websitesof incumbents when visiting websites

eDOTeDOT The “enhanced DOT” -- database and The “enhanced DOT” -- database and

software program developed by Economic software program developed by Economic Research Institute (ERI)Research Institute (ERI)

ERI began as compensation information ERI began as compensation information provider in 1987provider in 1987

Started the eDOT Skills Project in 2002 to Started the eDOT Skills Project in 2002 to collect data and update the DOTcollect data and update the DOT

In 2004, ERI purchased PAQ; PAQ In 2004, ERI purchased PAQ; PAQ operates eDOT Skills Project under a operates eDOT Skills Project under a license with ERIlicense with ERI

Database has two parts: archived DOT and Database has two parts: archived DOT and new eDOT which includes the old DOTnew eDOT which includes the old DOT

eDOTeDOT Includes 64 SCO characteristics + 35 Includes 64 SCO characteristics + 35

new factors such as keyboarding, new factors such as keyboarding, education, mental and cognitive factors, education, mental and cognitive factors, etc.etc.

Includes 20 measures from SSA’s Mental Includes 20 measures from SSA’s Mental Residual Capacity paperResidual Capacity paper

Various people contribute data: subject Various people contribute data: subject matter experts (job analysts trained in matter experts (job analysts trained in the use of the PAQ; voc rehab the use of the PAQ; voc rehab counselors; major disability carriers; counselors; major disability carriers; workers compensation analysts; and workers compensation analysts; and primarily incumbents visiting websites primarily incumbents visiting websites Career Builder, SalaryExpert, Career Builder, SalaryExpert, SalariesReview.SalariesReview.

eDOTeDOT Each data point has associated Each data point has associated

reliability, standard error and deviation reliability, standard error and deviation calculatedcalculated

1,000-1,500 job analyses done per year, 1,000-1,500 job analyses done per year, including those by incumbentsincluding those by incumbents

Over 1 million PAQ job analyses included Over 1 million PAQ job analyses included in eDOT (completed since 1974)in eDOT (completed since 1974)

Sample is one of convenienceSample is one of convenience Control for sampling error by using Control for sampling error by using

multiple sources of datamultiple sources of data

eDOTeDOT Differences between eDOT and DOTDifferences between eDOT and DOT

• Different rating scales, definitions, termsDifferent rating scales, definitions, terms• eDOT→ uses revised version of SIC which eDOT→ uses revised version of SIC which

it developed, called eSICit developed, called eSIC• eDOT→fluid database; DOT→fixedeDOT→fluid database; DOT→fixed• eDOT→convenience sample; DOT→attempt eDOT→convenience sample; DOT→attempt

to capture all jobs in the national economyto capture all jobs in the national economy• Electronic v. paperElectronic v. paper• PAQ→ interviewing worker, not always PAQ→ interviewing worker, not always

observing; DOT→observationobserving; DOT→observation

eDOTeDOT Examples of new jobs added to eDOTExamples of new jobs added to eDOT

computer sys admincomputer sys admin030.162-500030.162-500

call center repcall center rep 299.357-201299.357-201

asst. mgr retail storeasst. mgr retail store 185.167-505185.167-505

executive asstexecutive asst 169.167-911169.167-911

sales assistantsales assistant 209.562-800209.562-800

maint. helpermaint. helper 806.687-011806.687-011

CAD/CAM TechCAD/CAM Tech 017.262-519017.262-519

eDOTeDOT ERI/PAQ “masks” jobsERI/PAQ “masks” jobs Criteria for exclusion: Criteria for exclusion:

• job analyses over 15 years old job analyses over 15 years old • face validity (abalone diver)face validity (abalone diver)• 10 or less requests for the job10 or less requests for the job• not listed on job boards or salary not listed on job boards or salary

surveyssurveys• not mentioned on any loan applicationsnot mentioned on any loan applications• not on any labor/proxy/form 990not on any labor/proxy/form 990• not mentioned on Calif. state work not mentioned on Calif. state work

compcomp

eDOTeDOT Examples of “masked” jobs in eDOT:Examples of “masked” jobs in eDOT:

animal breederanimal breeder 410.161-010 410.161-010 military recruitermilitary recruiter 166.267-026166.267-026

pres., financial inst.pres., financial inst. 186.117-054186.117-054

police commissionerpolice commissioner 188.117-118188.117-118

feed blenderfeed blender 520.685-094520.685-094

collator operatorcollator operator 208.685-010208.685-010

eDOTeDOT As of January 2008, ERI hadAs of January 2008, ERI had

• Added 717 new occupationsAdded 717 new occupations• Removed 4,103 occupations (no/low Removed 4,103 occupations (no/low

frequency)frequency)• Verified the existence of 8,658 Verified the existence of 8,658

occupationsoccupations• Total of 9,375 occupations in eDOT, Total of 9,375 occupations in eDOT,

compared to 12,761 in the DOT and compared to 12,761 in the DOT and 812 in O*NET812 in O*NET

• As of July 2008, have added WF and As of July 2008, have added WF and MPSMS to all jobs in eDOTMPSMS to all jobs in eDOT

McDOT HistoryMcDOT History

McCroskey Dictionary of Occupational McCroskey Dictionary of Occupational TitlesTitles (McDOT) (McDOT)

Part of the McCroskey Vocational Part of the McCroskey Vocational Quotient System (MVQS)Quotient System (MVQS)

Based on VDAREBased on VDARE McDOT includes both the old DOT and McDOT includes both the old DOT and

O*NET. O*NET. Data has been fused from both Data has been fused from both

sources, identifying 24 most sources, identifying 24 most significant worker traitssignificant worker traits

UNUM: Project with UNUM: Project with eDOTeDOT Methodology for selecting eDOTMethodology for selecting eDOT

FindingsFindings

Future DirectionsFuture Directions

Issues with DOTIssues with DOTMiller et al. Study 1980Miller et al. Study 1980

Work, Jobs and Occupations: A Work, Jobs and Occupations: A Critical Review of the Dictionary Critical Review of the Dictionary of Occupational Titlesof Occupational Titles

Comprehensive review & evaluation of Comprehensive review & evaluation of the 1977, 4the 1977, 4thth Ed. DOT, conducted on Ed. DOT, conducted on behalf of the National Research behalf of the National Research Council, at the request of the Council, at the request of the Department of LaborDepartment of Labor


““The comprehensiveness, The comprehensiveness, reliability and accuracy of the reliability and accuracy of the DOT are in large part a function DOT are in large part a function of the data collection and of the data collection and analysis procedures used to analysis procedures used to produce it.” p.114produce it.” p.114


Data Collection/Methodology Issues:Data Collection/Methodology Issues:• Lack of written procedures on how Lack of written procedures on how

DOT produceDOT produce• Majority of principles used to create Majority of principles used to create

DOT were established in 1939 & DOT were established in 1939 & 19491949

• Sampling plan “complicated and Sampling plan “complicated and indirect,” did not include all jobs in indirect,” did not include all jobs in US economyUS economy


Data Collection/Methodology Issues:Data Collection/Methodology Issues:• Heavy orientation to manufacturing Heavy orientation to manufacturing

sector sector • Limited review of jobs requiring Limited review of jobs requiring

cognitive skills over physical skillscognitive skills over physical skills• Inadequate update schedule Inadequate update schedule


Found analysts often unable to Found analysts often unable to provide concrete explanation for how provide concrete explanation for how they rated worker traits besides they rated worker traits besides subjective means or past experiencesubjective means or past experience

Some occupations not reviewed at all; Some occupations not reviewed at all; others reviewed excessively: material others reviewed excessively: material handler → 652 job analyseshandler → 652 job analyses

1979 study random job titles found 1979 study random job titles found 81% of 481% of 4thth Ed. descriptions exactly the Ed. descriptions exactly the same as the 3same as the 3rdrd Edition Edition


Data Collection/Methodology Issues:Data Collection/Methodology Issues:• 16% occupational descriptions 16% occupational descriptions

completed without a single job completed without a single job analysisanalysis

• 29% based on one job analysis29% based on one job analysis• 19% based on two job analyses19% based on two job analyses• 36% based on three or more 36% based on three or more

analysesanalyses


Reliability/Validity Issues:Reliability/Validity Issues:• Validity: measurements of worker Validity: measurements of worker

traits/functions found to be fairly unreliable: traits/functions found to be fairly unreliable: “vague and ambiguously defined. Not readily “vague and ambiguously defined. Not readily apparent what the variables are intended to apparent what the variables are intended to measure” (p. 164)measure” (p. 164)

• Worker traits/functions developed in the Worker traits/functions developed in the 1950s—questionable validity for today’s labor 1950s—questionable validity for today’s labor marketmarket

• Question whether GED and SVP measure Question whether GED and SVP measure prestige or social status of occupationsprestige or social status of occupations


Reliability/Validity Issues:Reliability/Validity Issues:• GED scale validated against curriculum GED scale validated against curriculum

content, not validated for occupational content, not validated for occupational performanceperformance

• Working conditions & physical demands Working conditions & physical demands appear to be developed for unskilled, appear to be developed for unskilled, physical jobsphysical jobs

• Reliability: items are scored subjectively; Reliability: items are scored subjectively; raters had trouble assigning some factors, raters had trouble assigning some factors, particularly SVP and aptitudesparticularly SVP and aptitudes

• Reliabilities higher for manufacturing jobs Reliabilities higher for manufacturing jobs than for service jobsthan for service jobs

Issues with DOTIssues with DOT

Cain et al. (1983) study on reliabilities Cain et al. (1983) study on reliabilities of different ratings.of different ratings.• Two job descriptions per 24 occupations; Two job descriptions per 24 occupations;

experienced analysts rated the factorsexperienced analysts rated the factors• Found acceptable reliabilities for: data, Found acceptable reliabilities for: data,

people, GED reasoning, GED language people, GED reasoning, GED language and SVPand SVP

• Modest reliabilities for things, GED math, Modest reliabilities for things, GED math, strength factors and locationstrength factors and location

• Reliabilities higher for manufacturing jobsReliabilities higher for manufacturing jobs


Botterbusch (1993): DOT weaknesses:Botterbusch (1993): DOT weaknesses:• Data people things not actually a Data people things not actually a

hierarchy—data is, people is not and hierarchy—data is, people is not and things is two hierarchiesthings is two hierarchies

• GED not directly related to educationGED not directly related to education• SVP does not distinguish between SVP does not distinguish between

formal and informal trainingformal and informal training• Difficulty defining, using and Difficulty defining, using and

defending temperamentsdefending temperaments• Too many titlesToo many titles


Dunn & Growick (2000): weaknesses:Dunn & Growick (2000): weaknesses:• Failure to include variables such as org. Failure to include variables such as org.

setting and worker responsibility levelsetting and worker responsibility level• Low reliability of worker traitsLow reliability of worker traits• Redundant classification systemRedundant classification system• Skills that are psychometrically Skills that are psychometrically

questionable, such as GED and SVPquestionable, such as GED and SVP• Ambiguous definitions of worker traitsAmbiguous definitions of worker traits• Questionable validity of worker traitsQuestionable validity of worker traits


Harvey (Fine et al. 2004) addressed Harvey (Fine et al. 2004) addressed weaknesses of DOT:weaknesses of DOT:• Legal and psychometric Legal and psychometric

defensibility in the use of holistic defensibility in the use of holistic scales to rate worker traits—tend to scales to rate worker traits—tend to show low inter-rater reliability and show low inter-rater reliability and low discriminant validitylow discriminant validity

• Lack of reasonable update scheduleLack of reasonable update schedule


National Research Council (1999)National Research Council (1999)• ““Unwieldy size”Unwieldy size”• Growing differences between job Growing differences between job

descriptions and jobs as they descriptions and jobs as they actually occur in the labor market actually occur in the labor market todaytoday

• Too much detail in each definitionToo much detail in each definition• Jobs described by taskJobs described by task


National Research Council cont. National Research Council cont. (1999)(1999)• Lack of information on factors such Lack of information on factors such

as skills, abilities, and knowledge as skills, abilities, and knowledge requirements that are either not requirements that are either not collected or not reportedcollected or not reported

• Little or no reported information on Little or no reported information on cognitive factorscognitive factors

• Expense of updatingExpense of updating• Difficulty of linking it with other Difficulty of linking it with other

databasesdatabases


APDOT (1993): need to reinvent the APDOT (1993): need to reinvent the DOTDOT• Representative of economyRepresentative of economy• Develop new JA methodology: found Develop new JA methodology: found

existing ones to be lacking, existing ones to be lacking, particularly with cognitive skillsparticularly with cognitive skills

• Create one better at job matching, Create one better at job matching, skills transferskills transfer

•


1980 Miller et al. study issued several 1980 Miller et al. study issued several conclusions and a number of recommendations conclusions and a number of recommendations (p.214)(p.214)

Conclusion #1Conclusion #1 – “There is a strong and – “There is a strong and continuing need both within and outside the continuing need both within and outside the U.S. Employment Service for the kind of U.S. Employment Service for the kind of information provided by the Dictionary of information provided by the Dictionary of Occupational Titles and certain other products Occupational Titles and certain other products based on it.” based on it.”

Conclusion #2Conclusion #2 – “Substantial improvements in – “Substantial improvements in the procedures and products of the the procedures and products of the occupational analysis program are required in occupational analysis program are required in order to meet the national need for order to meet the national need for occupational information.”occupational information.”

Issues with O*NETIssues with O*NET

“…“…O*NET as currently implemented O*NET as currently implemented arguably fails to provide (a) occupational arguably fails to provide (a) occupational data expressed at the appropriate level of data expressed at the appropriate level of detaildetail and and defensibilitydefensibility needed for many needed for many functions and (b) occupational title functions and (b) occupational title taxonomy that is sufficiently taxonomy that is sufficiently detaileddetailed and and phrased at the desired level of analysis” phrased at the desired level of analysis” (Fine et al., 2004, p1).(Fine et al., 2004, p1).

Depth of material in O*NET: telling Depth of material in O*NET: telling someone which state you live in but not someone which state you live in but not giving your address (ERI)giving your address (ERI)


Terminology and measurement scales Terminology and measurement scales are poorly definedare poorly defined

Does not provide correct type of Does not provide correct type of information on physical demands, information on physical demands, training requirements and other training requirements and other information essential for TSAinformation essential for TSA

Job demands difficult to measure (e.g. Job demands difficult to measure (e.g. cognitive)cognitive)

Use of numerical scales over interval Use of numerical scales over interval scalesscales


Completed by incumbents, except for Completed by incumbents, except for abilities sectionabilities section

Problem with incomplete survey Problem with incomplete survey instrumentsinstruments

Uses mean data rather than modal Uses mean data rather than modal data as found in the DOTdata as found in the DOT

On the + Side: O*NETOn the + Side: O*NET

Many new variables to describe work-Many new variables to describe work-related characteristics and behaviorrelated characteristics and behavior

Includes relevant aspects of work not Includes relevant aspects of work not in the DOT, i.e. work contextin the DOT, i.e. work context

Expanded knowledge requirements, Expanded knowledge requirements, training and preparation needstraining and preparation needs

IODC Evaluation of eDOTIODC Evaluation of eDOT

ERI provided complimentary eDOT ERI provided complimentary eDOT software plus two training sessionssoftware plus two training sessions

Met with ERI staff at 2007 Forensic Met with ERI staff at 2007 Forensic Conference to discuss issues/concernsConference to discuss issues/concerns

ERI staff available to answer ERI staff available to answer questions throughout processquestions throughout process

Separate training provided by user of Separate training provided by user of eDOT at UNUMeDOT at UNUM

Provided beta versions of OccuBrowse Provided beta versions of OccuBrowse database with integrated eDOT database with integrated eDOT


Intention to update DOT positiveIntention to update DOT positive Creative in data gathering processesCreative in data gathering processes Inclusion of cognitive elementsInclusion of cognitive elements Staffed with I/O psychologistsStaffed with I/O psychologists Presentation of raw data and data Presentation of raw data and data

sources for professional studysources for professional study Separation of physical demandsSeparation of physical demands


New ratings gathered from New ratings gathered from convenience sample convenience sample

Industries and occupations do not Industries and occupations do not always match upalways match up

Coding of occupations different than Coding of occupations different than DOT DOT

Uses mean trait ratings rather than Uses mean trait ratings rather than mode used in DOT mode used in DOT

Method for removing occupations Method for removing occupations unsatisfactoryunsatisfactory


Method of data aggregation to define Method of data aggregation to define new occupations are unclearnew occupations are unclear

Method used to translate PAQ data Method used to translate PAQ data into DOT terms is unclearinto DOT terms is unclear

Several listings for same job--Several listings for same job--confusingconfusing

Unclear and excessive number of Unclear and excessive number of rating scalesrating scales

Based on the flawed DOTBased on the flawed DOT

IODC Evaluation of IODC Evaluation of McDOTMcDOT

IODC provided with complementary IODC provided with complementary software software

One training session One training session Vocationology available for questions, Vocationology available for questions,

issues throughout processissues throughout process


The new job analysis information in The new job analysis information in McDOT comes directly from O*NET—McDOT comes directly from O*NET—positive and negativepositive and negative

Analyses of inter-correlations among Analyses of inter-correlations among various worker trait characteristicsvarious worker trait characteristics

Reliability and validity dataReliability and validity data No independent job analyses were No independent job analyses were

conducted by Vocationology, Inc.conducted by Vocationology, Inc.


Since O*NET was initially derived Since O*NET was initially derived from DOT data, O*NET is a from DOT data, O*NET is a confounded data source as an confounded data source as an “update” to the DOT.“update” to the DOT.

Incorporates all problems of both DOT Incorporates all problems of both DOT and O*NETand O*NET

SSA has reported it will not use a SSA has reported it will not use a database with O*NET in itdatabase with O*NET in it

Difficult for many to understand or Difficult for many to understand or explain to judges, juries, attorneys, explain to judges, juries, attorneys, etc.etc.

IODC FindingsIODC Findings

IODC was unable to identify and IODC was unable to identify and endorse a ready-to-use database that endorse a ready-to-use database that fits members needs and needs of SSAfits members needs and needs of SSA

eDOT has potential to be that eDOT has potential to be that database, but many methodological database, but many methodological problems which need to be solved problems which need to be solved before it is usable for transferable before it is usable for transferable skills analysis and in court settingsskills analysis and in court settings

IODC recommends continued use of IODC recommends continued use of DOT until better product is availableDOT until better product is available

Moving toward a better Moving toward a better DOTDOT

Job Analysis Issues:Job Analysis Issues:• Identification of an agreed-upon job Identification of an agreed-upon job

analysis format: reliable & valid, analysis format: reliable & valid, standardized formatstandardized format

• Re-evaluation and redefinition of Re-evaluation and redefinition of worker characteristics; e.g. what worker characteristics; e.g. what constitutes a “physical demand”constitutes a “physical demand”

• Develop improved definitions and Develop improved definitions and indicators for attributes; scales for indicators for attributes; scales for eacheach

• Use of decomposed rather than Use of decomposed rather than holistic ratingsholistic ratings


Job Analysis Issues:Job Analysis Issues:• Expanded physical demand ratings: Expanded physical demand ratings:

sitting, standing, walking, reaching sitting, standing, walking, reaching (above shoulder v. forward v. (above shoulder v. forward v. downward), vehicle operation, lift, downward), vehicle operation, lift, carry, push, pull, use of controls carry, push, pull, use of controls (hands, feet), bilateral/unilateral upper (hands, feet), bilateral/unilateral upper extremity use, exposure to chemical extremity use, exposure to chemical agents, biological hazards, latex, agents, biological hazards, latex, animals, potential food allergens, animals, potential food allergens, photic triggers (seizure potential)photic triggers (seizure potential)


Job Analysis Issues:Job Analysis Issues:• Addition of “basic” skills including: Addition of “basic” skills including:

keyboarding, technology use keyboarding, technology use (computer user, familiarity with (computer user, familiarity with multiple applications, train others, multiple applications, train others, troubleshooting, etc.)troubleshooting, etc.)

• Addition of attributes, including Addition of attributes, including cognitive requirements; deletion of cognitive requirements; deletion of othersothers


Job Analysis Issues:Job Analysis Issues:• Preferred personal qualities of Preferred personal qualities of

workers for successful performance: workers for successful performance: attitude, initiative, persistence, attitude, initiative, persistence, resourcefulness, honesty, flexibility, resourcefulness, honesty, flexibility, team orientation, communication team orientation, communication skills, tact, organization, leadership, skills, tact, organization, leadership, efficiency, versatility, reliability, efficiency, versatility, reliability, quality orientation, attentiveness, quality orientation, attentiveness, etc.etc.


Job Analysis Issues:Job Analysis Issues:• Acceptable methods of preparation Acceptable methods of preparation

for entry into occupations: formal for entry into occupations: formal education, vo-tech, apprenticeship, education, vo-tech, apprenticeship, in-service training, OJT, in-service training, OJT, certification, licensure, registration, certification, licensure, registration, prior experienceprior experience

• Barriers to hiring for a specific Barriers to hiring for a specific occupation: criminal history, occupation: criminal history, appearance, personal hygiene appearance, personal hygiene habits, monocular vision, etc. habits, monocular vision, etc.


Data Collection/Methodology:Data Collection/Methodology:• Comprehensive, multi-level stratified Comprehensive, multi-level stratified

sampling plansampling plan• Audit to identify where occupations Audit to identify where occupations

are found within businesses are found within businesses /industries of all sizes (e.g. small /industries of all sizes (e.g. small employers with <25 employees, self-employers with <25 employees, self-employment)employment)

• Use only trained/qualified Use only trained/qualified professionals as job analysts professionals as job analysts


Data Collection/Methodology:Data Collection/Methodology:• Determine how job analysts would Determine how job analysts would

be trained and retrained: what, who, be trained and retrained: what, who, where, whenwhere, when

• Identify whether JAs would be done Identify whether JAs would be done solely by resurrected Field Offices, solely by resurrected Field Offices, public sector or with help from public sector or with help from private. private.

• Improvement in technical quality of Improvement in technical quality of data and methodologies used, must data and methodologies used, must be ongoing in order to ensure its be ongoing in order to ensure its defensibility defensibility


Data Collection/Methodology:Data Collection/Methodology:• Coding methodology used must Coding methodology used must

make sensemake sense• 4 raters produces good reliability; 4 4 raters produces good reliability; 4

raters should be goalraters should be goal• Avoid use of incumbent ratings due Avoid use of incumbent ratings due

to reliability problems with this datato reliability problems with this data• Online system for input of job Online system for input of job

analysisanalysis


Data Collection/MethodologyData Collection/Methodology• Ensure database reflects the Ensure database reflects the

economy/labor market with at least 90% economy/labor market with at least 90% (preferred 95%) confidence – tied to (preferred 95%) confidence – tied to number of analyses performed and number of analyses performed and variability in results obtainedvariability in results obtained

• Attributes must have multiple measuresAttributes must have multiple measures• Appropriate scaling for each attribute Appropriate scaling for each attribute

and sub-measures must make sense and sub-measures must make sense


Data Collection/MethodologyData Collection/Methodology• Proper instrumentation/equipment Proper instrumentation/equipment

must be available to conduct an must be available to conduct an objective, measurable rather than objective, measurable rather than strictly observational job analysisstrictly observational job analysis

• Total number and range of Total number and range of attributes should be limited to what attributes should be limited to what an analyst can handle without an analyst can handle without diminishing quality of data being diminishing quality of data being gatheredgathered


Data Collection/MethodologyData Collection/Methodology • Identify variables needed in order to Identify variables needed in order to

do a TSA-job match; once identified, do a TSA-job match; once identified, develop scales, use accepted develop scales, use accepted psychometric practices—this would psychometric practices—this would increase reliabilityincrease reliability


Use of Data/DOTUse of Data/DOT• Methodology needs to be explained Methodology needs to be explained

in as simple terms as possible for in as simple terms as possible for ease of customer use ease of customer use

• Processes & methodologies, as well Processes & methodologies, as well as results of data usage, must be as results of data usage, must be easily explainable to ALJs/judges, easily explainable to ALJs/judges, juries, attorneys, etc.juries, attorneys, etc.

• Software used to access data and Software used to access data and perform TSAs, etc., should be perform TSAs, etc., should be simplified wherever possible to simplified wherever possible to reduce errors/improve reduce errors/improve understandabilityunderstandability


Use of Data/DOTUse of Data/DOT• Needs to be available as stand-alone Needs to be available as stand-alone

product, not solely as part of a product, not solely as part of a transferable skills analysis producttransferable skills analysis product

• Allow end-users to search, compare, Allow end-users to search, compare, and retrieve informationand retrieve information

• Development of crosswalks should Development of crosswalks should be well-explainedbe well-explained


Use of Data/DOTUse of Data/DOT• Capable of generating printed Capable of generating printed

reportsreports• Data available to end-users in Data available to end-users in

variety of formats, including online variety of formats, including online and in print; particularly important and in print; particularly important that data not require Internet that data not require Internet connection so it can be accessed connection so it can be accessed during Social Security hearingsduring Social Security hearings


Use of Data/DOTUse of Data/DOT• Author should have vocational Author should have vocational

rehabilitation person(s) on staff for rehabilitation person(s) on staff for development phase and customer development phase and customer supportsupport


Updates:Updates:• Continuously updatedContinuously updated• Monitor changes in labor market; Monitor changes in labor market;

reflect changes within the databasereflect changes within the database• Work closely with OES Long Term Work closely with OES Long Term

Projections Survey to learn of new Projections Survey to learn of new emerging occupations emerging occupations

• Sufficient funding must be provided Sufficient funding must be provided to develop improved database and to develop improved database and ensure maintenance into the futureensure maintenance into the future


Integration:Integration:• Method for integrating new DOT Method for integrating new DOT

with existing occupational with existing occupational classification systems (SOC, O*NET, classification systems (SOC, O*NET, NAICS, etc.) NAICS, etc.)

• Decisions must be made on other Decisions must be made on other Labor Market Information Labor Market Information databases that should be tied into databases that should be tied into an improved occupational database an improved occupational database to replace the D.O.T. to replace the D.O.T.

The IODC CommitteeThe IODC Committee

Angela Heitzman, Angela Heitzman, MA, CRC, CLCP, MSCCMA, CRC, CLCP, MSCC John Meltzer, John Meltzer, MS, CRC, CDMS, LPCMS, CRC, CDMS, LPC Sonia Paquette, Sonia Paquette, OTD, OTR/L, CPE, ABVE-DOTD, OTR/L, CPE, ABVE-D Gerald Schneck, Gerald Schneck, PhD, CRC-MAC, FVE, NCCPhD, CRC-MAC, FVE, NCC Jeffrey Truthan, Jeffrey Truthan, MS, CVEMS, CVE With assistance from Kelly Beery-Marsiano, MEd, With assistance from Kelly Beery-Marsiano, MEd,

CRC; Betty Morris, MS, CRC; G. Shannon CRC; Betty Morris, MS, CRC; G. Shannon O’Kelley, MEd, CRC; and Sharon Shou, CRC, O’Kelley, MEd, CRC; and Sharon Shou, CRC, LCPC, ABVE, all of UNUMLCPC, ABVE, all of UNUM

ReferencesReferences

Botterbusch, K.F. (1993, Fall). Suggestions for Botterbusch, K.F. (1993, Fall). Suggestions for revisions in the dictionary of occupational titles. revisions in the dictionary of occupational titles. Vocational Evaluation and Work Adjustment Vocational Evaluation and Work Adjustment BulletinBulletin, 101-110., 101-110.

Cain, P.S. & Green, B.F. (1983). Reliabilities of Cain, P.S. & Green, B.F. (1983). Reliabilities of selected ratings available from the selected ratings available from the Dictionary of Dictionary of Occupational Titles. Journal of Applied Occupational Titles. Journal of Applied Psychology, 68Psychology, 68(1), 155-165.(1), 155-165.

Dunn, P.L. & Growick, B.S. (2000). Transferable Dunn, P.L. & Growick, B.S. (2000). Transferable skills analysis in vocational rehabilitation: skills analysis in vocational rehabilitation: Historical foundations, current status, and Historical foundations, current status, and future trends. future trends. Journal of Vocational Journal of Vocational Rehabilitation, 14Rehabilitation, 14, 79-87., 79-87.


Economic Research Institute. (April 2008). Economic Research Institute. (April 2008). Methodology/Disclaimer: Occupational Assessor™ Methodology/Disclaimer: Occupational Assessor™ and Survey (OA/OA+) -- eDOT®, Software and and Survey (OA/OA+) -- eDOT®, Software and Databases (Professional and Consultant Editions).Databases (Professional and Consultant Editions). Retrieved from: Retrieved from: www.paq.com/pdf/e-dot-methodology.pdfwww.paq.com/pdf/e-dot-methodology.pdf. .

Fine, S.A., Harvey, R.J. & Crenshaw, S.F. (2004, Fine, S.A., Harvey, R.J. & Crenshaw, S.F. (2004, April). FJA strategies for addressing O*NET April). FJA strategies for addressing O*NET limitations in a post-DOT environment. In limitations in a post-DOT environment. In Fleischman, E.A. (Chair), Fleischman, E.A. (Chair), Things, Data, and Things, Data, and People: Fifty Years of a Seminal TheoryPeople: Fifty Years of a Seminal Theory. . Symposium presented at the Annual Conference Symposium presented at the Annual Conference of the Society for Industrial and Organizational of the Society for Industrial and Organizational Psychology, Chicago.Psychology, Chicago.


Mariani, M. (1999, Spring). Replace with a Mariani, M. (1999, Spring). Replace with a database: O*NET replaces the Dictionary database: O*NET replaces the Dictionary of Occupational Titles. of Occupational Titles. Occupational Occupational Outlook Quarterly,Outlook Quarterly, 43(1), 3-9. 43(1), 3-9.

McCroskey, B.J. (2008). McCroskey, B.J. (2008). Volcano 8.0.Volcano 8.0. Brooklyn Park, MN: Vocationology., Inc.Brooklyn Park, MN: Vocationology., Inc.

Miller, A.R., Treiman, D.J., Cain, P.S. & Miller, A.R., Treiman, D.J., Cain, P.S. & Roos, P.A. (Eds). (1980). Roos, P.A. (Eds). (1980). Work, Jobs and Work, Jobs and Occupations: A Critical Review of the Occupations: A Critical Review of the Dictionary of Occupational TitlesDictionary of Occupational Titles. . Washington, DC: National Academy Press.Washington, DC: National Academy Press.


National Research Council. (1999). National Research Council. (1999). The Changing The Changing Nature of Work: Implications for Occupational Nature of Work: Implications for Occupational Analysis.Analysis. Committee on Techniques for the Committee on Techniques for the Enhancement of Human Performance: Enhancement of Human Performance: Occupational Analysis, Commission on Occupational Analysis, Commission on Behavioral and Social Sciences and Education. Behavioral and Social Sciences and Education. National Academy of Sciences: Washington, DC.National Academy of Sciences: Washington, DC.

United States Department of Labor. (1991). United States Department of Labor. (1991). Dictionary of Occupational Titles (4th Ed.).Dictionary of Occupational Titles (4th Ed.). Washington, DC: United States Government Washington, DC: United States Government Printing Office.Printing Office.

United States Department of Labor. (1991). United States Department of Labor. (1991). The The Revised Handbook for Analyzing JobsRevised Handbook for Analyzing Jobs. . Washington, DC: United States Government Washington, DC: United States Government Printing Office.Printing Office.


United States Department of Labor. (1993). United States Department of Labor. (1993). Selected Characteristics of Occupations Selected Characteristics of Occupations Defined in the Revised Dictionary of Defined in the Revised Dictionary of Occupational TitlesOccupational Titles. Washington, DC: . Washington, DC: United States Government Printing Office. United States Government Printing Office.

U.S. Department of Labor (1993). U.S. Department of Labor (1993). The New The New DOT: A Database of Occupational Titles for DOT: A Database of Occupational Titles for the 21st-Century.the 21st-Century. Advisory Panel for the Advisory Panel for the Dictionary of Occupational Titles (APDOT).Dictionary of Occupational Titles (APDOT).

United States Department of Labor. (2008). United States Department of Labor. (2008). The O*NET Content ModelThe O*NET Content Model. Retrieved . Retrieved from from http://www.onetcenter.org/dl_files/Contenthttp://www.onetcenter.org/dl_files/ContentModel_DetailedDesc.pdfModel_DetailedDesc.pdf. .

Documents

Whats Up DOT? IARP Occupational Database Committee IARP Forensic Conference Hyatt Regency Bonaventure Weston, Florida October 31, 2008