Improving Methods of Resident SelectionJ.D. Prager, C.M. Myer IV, K. Hayes, C.M. Myer III, M.L. Pensak

Cincinnati Children’s Hospital Medical Center

ResultsWPS job analysis generated 10 competencies and specific behavioral-based interview questions designed to explore those competencies (Table to right). Applicant scores on these questions were evaluated during the department ranking meeting. These scores had high correlation with the applicant’s rank position (Pearson correlation coefficient for rank number and mean interview score was 0.80 (p<0.0001)). See figure below.

AbstractObjectives: Applying the concept of the ACGME general competencies, it is possible to define the essential job objectives and competencies of a junior otolaryngology resident. The objective of this study is to incorporate commercially available tools of business in the identification of competencies specific to the junior otolaryngology resident and develop behavioral-based interview questions and techniques designed to identify these qualities in candidates for residency.

Study Design: Institution of a pilot program involving a focus group within an otolaryngology department, a professional development consultant, commercial business software for occupational analysis and personnel selection, and an interview technique training seminar for faculty and residents.

Methods: In coordination with a university-based professional development consultant, a formal job analysis was conducted to define the job objectives and competencies of a junior otolaryngology resident. These results were used to generate behavioral-based interview questions for use in the resident selection process. All interviewing faculty and residents were trained in behavioral-based interviewing.

Results: Occupational objectives for the junior resident position specific to a particular university department of otolaryngology were identified. Additionally, the essential skills, areas of knowledge and competencies were identified. Behavioral-based questions specific to the competencies were created and incorporated into the current resident selection interview.

Conclusions: Using tools of occupational analysis and personnel selection, a list of job objectives and competencies for the junior otolaryngology resident can be created. Using these results, behavioral-based interviews may be implemented to complement traditional interviews with the ultimate goal of improving candidate selection.

IntroductionMultiple factors combine to challenge the modern otolaryngology training program, including resident attrition (failure to complete training) and increasing pressure to utilize limited resources efficiently. The resident selection process offers an opportunity to address these challenges through selection of the most qualified applicants who best-fit the training program and are most likely to complete training successfully. However, the traditional selection process relies on past academic data and traditional interviews which are not consistently valid and reliable predictors of success during residency. This study emerged from an effort within one academic training program to improve the resident selection process by using established, rigorous and reliable methods not in frequent use within medicine: job analysis and behavioral-based interviewing.

Job analysis techniques have as their goal the identification of job-specific objectives, essential and desirable knowledge and skills, and competencies that characterize a specific job and its effective performer. Job analysis has been used in business, and less frequently within medicine, for decades. Behavioral-based interviewing has demonstrated improved reliability and validity in comparison to traditional interviewing techniques in predicting job performance.

DiscussionUsing commercially available software and the skills of a professional development consultant, a job analysis for the position of junior otolaryngology resident may be conducted to determine job-specific objectives, essential and desirable knowledge and skills, and competencies. Behavioral-based interview questions specifically intended investigate these criteria may then be created and implemented. This pilot program demonstrates that the above measures may be quickly and efficiently implemented to improve the process of resident selection.

Methods1. Job Analysis

With the aid of a university-based professional development consultant, we performed a job analysis for the role of junior otolaryngology resident using the SHL Work Profiling System (SHL Group, UK). The Work Profiling System (WPS) requires that a group of people familiar with the job (usually 4-8) define the main purpose of the job and the top ten job objectives. The group then ranks (in terms of time and importance) various tasks, activities, skills and knowledge necessary to achieve the main purpose and objectives they defined. WPS software then analyzes the information collected using psychometric principles and generates reports presenting the critical aspects of the job and the human attributes necessary to achieve them. We have defined these as the competencies of the junior otolaryngology resident. WPS also generated behavioral-based interview questions specific to these competencies.

Compared with other people, how energetic are you? How would you rate your endurance?If your job did not allow you to be physically active, how frustrated would you be?How easy/difficult do you find it to remain seated or standing for long periods?What sort of physical training do you do? How do you organize/plan it? How effective is this?

• Making a fast physical reaction• Exercising fine eye-hand coordination• Exercising fine manual dexterity

10. Physical Activities

With what sort of computers are you familiar? How would you rate your level of expertise?How useful do you find new technologies? What difficulties arise from new technologies?

• Using precision tools or instruments• Using a computer for information storage or retrieval

9. Using Tools/Machinery

To what extent have you been involved in assessing…? (people/information/products/objects)What difficulties did you face? How did you overcome them? How did you select criteria for assessment? How was your assessment received by others?

• Evaluating alternatives prior to choice8. Assessing/Evaluating

Give an example of your having to motivate people. How did you go about it? What response did you get? What difficulties did you face? What evidence was there of success?How do you feel your style of motivating people compares with others? More or less effective?What do you think is the best way of motivating people?

• Persuading an individual to carry out an unappealing task• Providing reassurance for those suffering anxiety• Putting people at ease in stressful situations

7. Motivating

What experience do you have carrying out interviews. What are your strengths/weaknesses?What do you think are the key skills for effective interviewing?Describe a recent interview you conducted. What was the objective? Were you able to achieve this?

• Discussing issues for clarification or explanation• Making a verbal report to a supervisor or manager• Answering critical questions about activities• Providing spoken information about a problem/issue• Interviewing formally to establish facts• Personally dictating notes or correspondence• Providing written information about a problem/issue

6. InformingDiscussingInterviewing

Describe a situation in which you failed to persuade someone to accept your view. Why didn’t it work?Name a team in which you function. How persuasive are you compared with other members? In what ways?Describe a recent negotiation in which you played a part. What was your contribution? Was it effective?

• Offering professional advice• Summarizing salient points in debate or disagreement• Making a spoken case for a course of action

5. Influencing/Advising

Describe a recent research project of yours. If you could start again, what would you do differently?What do you see to be the main stages of a research project?To what extent do you keep up-to-date in your field? How do you do this? What other things could you do?Have you ever found your expertise lacking? What action did you take?

• Learning new systems, methods or processes• Keeping abreast of developments in a specialist field

4. Learning/Researching

Give me an example of when you have had to analyze verbal or numerical information. What was the purpose of this analysis? What conclusions were you able to draw? To what extent were you under time pressure? Describe an occasion of when you diagnosed a problem which others had not been able to see. What were the outcomes? To what extent to people come to you for assistance with similar problems?Describe an example of when you perceived an opportunity through analysis which others had not been able to see. What process did you use?

• Diagnosing physical disorders• Analyzing numerical information• Analyzing written information

3. Analyzing/Diagnosing

To what extent are you allowed to make decisions in your current job? Is this enough/too much/not enough?Describe a recent important decision you made. If you had the chance again, would you do the same?Describe a team decision to which you recently contributed. What was your contribution? To what extent was your contribution an influence on the decision?What do you think are the key elements in making a good decision?Tell me about a poor decision you have made. What should you have done differently?

• Deciding a course of action in conjunction with others• Making quick decisions under time pressure• Mentally reacting quickly to emergency situations

2. Deciding

To what extent does your job rely on the senses? How sharp would you rate each of these senses?Describe an aspect of your job which requires careful, accurate vision. How often would you make a mistake by not looking carefully enough? How serious would such an effect be? Probe similarly for the other senses. Describe a time when you had to use multiple senses to make a decision. Tell me about a time when a person’s actions did not match their words. What did you do?

• Following logical instructions to operate a machine• Listening to verbal instructions from superiors• Following written instructions• Listening to verbal reports from subordinates• Making fine visual discrimination

1. Taking information from the senses

Behavioral-based QuestionsExamplesEssential Tasks/Competencies

ReferencesJanz T, Hellervik L, Gilmore DC. Behavior Description Interviewing. Boston, USA: Allyn and Bacon, Inc.; 1986.Wade, T. P., Andrus, C. H., and Kaminski, D. L. Evaluations of surgery resident performance correlate with

success in board examinations. Surgery. 1993; 113(6): 644-8.Calhoun KH, Hokanson JA, Bailey BJ. Predictors of residency performance: a follow –up study. Otolaryngol Head

Neck Surg. 1997; 116:647-51.Veloski J, Herman MW, Gonella JS, Zeleznik C, Kellow WF. Relationships between performance in medical school

and first postgraduate year. J Med Educ. 1979; 54:900-16.Daly KA, Levine SC, Adams GL. Predictors for resident success in otolaryngology. J Am Coll Surg. 2006;

202:649-54. Gardner B. A multivariate computer analysis of students’ performance as a predictor of performance as a surgical

intern. J Surg Res. 1972; 12:216-219.Wood PS, Smith WL, Altmaier EM, Tarico VS, Franken EA. A prospective study of cognitive and noncognitive

selection criteria as predictors of resident performance. Invest Radiol. 1990; 25:855-859.Warrick SS, Crumrine RS. Predictors of success in an anesthesiology residency. Acad. Med. 1986; 61:591-595.Flanagan JC. The Critical Incident Technique. Psych Bull. 1954; 51:327-58. Tarico V, Smith WL, Altmaier E, Franken EA, VanVelzen D. Critical incident interviewing in evaluation of resident

performance. Radiology. 1984; 152:327-9.Altmaier EM, From RP, Pearson KS, Gorbatenko-Roth KG, Ugolini KA. A prospective study to select and evaluate

anesthesiology residents: phase I, the critical incident technique. J Clin Anesth. 1997; 9:629-36. Blouin D, Dagnone JD. Performance critieria for emergency medicine: a job analysis. CJEM. 2008; 10(6):539-44. Komives E, Weiss ST, Rosa RM: The applicant interview as a predictor of resident performance. J Med Educ.

1984; 59:425-6. Orpen C. Patterned Behavior Interviews Versus Unstructured Interviews: A Comparative Validity Study. J Applied

Psych. 1985; 70(4):774-776. Altmaier EM, Smith WL, O’Halloran CM, Franken EA. The predictive utility of behavior-based interviewing

compared with traditional interviewing in the selection of radiology residents. Invest Rad. 1992; 27(5):385-9.Campion MA, Campion JE, Hudson Jr. PJ. Structured interviewing: a note on incremental validity and alternative

question types. J Applied Psych. 1994; 79(6):998-1002. Janz T. Initial comparisons of patterned behavior-based interviews versus unstructured interviews. J Applied

Psych. 1982; 67(5):577-580. Motowidlo SJ, Carter GW, Dunnette MD, Tippins N, Werner S, Burnett JR, Vaughan MJ. Studies of the structured

behavioral interview. J Applied Psych. 1992; 77(5):571-587.Easdown LJ, Castro PL, Shinkle EP, Small L, Algren J. The behavioral interview, a method to evaluate ACGME

competencies in resident selection: a pilot project. JEPM. 2005; 7(1):1-10.Neitzschman HR, Neitzschman LH, Dowling A. Key Component of Resident Selection: The Semistructured

Conversation. Acad Radiol. 2002; 9:1423- 1429. Edwards JC, Currie ML, Wade TP, Kaminski DL. Surgery resident selection process: a critical incident study. Eval

Health Prof. 1993; 16(1):73-86.Tarico VS, Altmaier EM, Smith WL, Franken Jr. EA, Berbaum KS. Development and validation of an

accomplishment interview for radiology residents. J Med Educ. 1986; 61:845-847.

Methods cont.2. Behavioral-Based Interviewing

A 3.5 hour training session was held to familiarize faculty and residents with behavioral-based interviewing principles and techniques. The consultant and the program director selected 4 competencies to be explored during resident-applicant interviews in the fall of 2009. Behavioral-based interview questions specific to these competencies were used and applicants answers were scored on a scale of 1 to 5.

ConclusionThe resident selection process may be improved quickly and efficiently by implementing job analysis and interview methods with improved reliability and validity compared to traditional selection methods. Long term follow-up of our residents is necessary to verify the internal validity of this project.