IE 656B Modeling Healthcare Systemshttp://web.ics.purdue.edu/~duffy/index656B

Vincent G. Duffy, Ph.D.Associate Professor

Regenstrief Center for Healthcare Engineering

School of Industrial Engineering and Department of Agricultural &

Biological EngineeringPurdue University

Monday, January 28, 2008 Evaluating Research – List of 10 Ways…

10 ways to look at a research paper 10 ways to look at a research paper systematically for critical analysissystematically for critical analysis 1. research ideas/question -what is the purpose? 2. what background support? - literature review 3. theoretical basis for analyzing question/hypothesis? 4. applicability-practical contribution? 5. theoretical contribution? 6. appropriate methodologies for carrying out study?

(determining variables, data collection, method used to test hypotheses, validity of measures and reliability)

7. appropriate statistical analyses and assumptions? 8. presentation of results: what do they really mean? 9. conclusions drawn: are they reasonable? 10. future work/research directions: any possibilities?

10 ways to look at a research paper 10 ways to look at a research paper systematically for critical analysissystematically for critical analysis 1. research ideas/question -what is the purpose? Rather than assessment by oral exam (1st paragraph),

research becomes a possible solution for the problem (behavior of team can not be assessed crisis).

Test effectiveness of training method. Continuation.

Previously – recall of incidents. Prefer standard and measure.

Develop ratings. Validate that methodology (videotape) can be used.

Alternate methods for considering Inter-rater reliability considered

10 ways to look at a research paper 10 ways to look at a research paper systematically for critical analysissystematically for critical analysis 2. what background support? - literature review Important because educational/clinical

experience/performance in critical incident – efficiency can affect overall health of patient.

(what are literature/refs. to support ‘why objective is important’)

p.9 measure two …. (ref. 18) Difficult w/out simulating – need evaluation tools

(p.9 ref. 10, 18-20) Also similar in aviation. (see also refs. 12-17) Other literature support could be to justify the

methodologies used, or the theoretical foundations.

10 ways to look at a research paper 10 ways to look at a research paper systematically for critical analysissystematically for critical analysis 3. theoretical basis for analyzing question/hypothesis?

That is the difference between lit review & theoretical foundation?

What are the hypotheses? No definite technique to measure…

“However, didn’t find why we need assessment…” (based on p.16) Rationale may be related to

application or may contribute to crisis management research.

What do you mean…theoretical foundations…? Also similar in aviation & use of simulations – for video tapes –

theoretical support for methodologies used. Support may be embedded in the statement (col. 1 p.9) “…these crisis

management behaviors in a ‘paradigm’ (model) we call ACRM (ref. 18-20).

10 ways to look at a research paper 10 ways to look at a research paper systematically for critical analysissystematically for critical analysis 4. applicability-practical contribution? Clearly this does fit as a practical contribution/application for the research.

“These performance assessment tools might be useful for educational research or for tracking a residents progress…”

“However, didn’t find why we need assessment…” (based on p.16) Rationale may be related to application or may contribute to

crisis management research. ‘…may contribute’ is not yet a practical contribution (has potential to

contribute) As we read…let’s focus on ‘evidence’ that suggests the method or finding

‘adopted’ by someone or org. What if authors adopted?

Partially ok… Trouble…kind of like referring to own ‘paradigm’. (good?) maybe….more information is needed… Better when the theory or application can be externally tested or

validated.

10 ways to look at a research paper 10 ways to look at a research paper systematically for critical analysissystematically for critical analysis 5. theoretical contribution? Maybe…

These performance assessment tools might be useful for educational research or for tracking a residents progress…

Depends on the nature of the contribution/how useful to ‘educational research’.

Can’t tell from abstract alone.. Can look at ….conclusions/discussion to get

further insight. “However, didn’t find why we need assessment…”

(based on p.16) Rationale may be related to application or may contribute to crisis management research.

10 ways to look at a research paper 10 ways to look at a research paper systematically for critical analysissystematically for critical analysis 6. appropriate methodologies for carrying out study?

(determining variables, data collection, method used to test hypotheses, validity of measures and reliability)

p.11 variability between ratings can not be considered a true measure of the true interrater reliability of the rating scales. ..

Yes. Based on theoretical support for methodologies presented on p.9.

p.13 (top of discussion) Justifying method/videotape… for analysis…

10 ways to look at a research paper 10 ways to look at a research paper systematically for critical analysissystematically for critical analysis 7. appropriate statistical analyses and assumptions?

Are these analyses appropriate (mapped well) to answer the questions/objective/hypotheses based on the data that is collected (for the variables that are identified/measured)?

ANOVA to check rate of variability as well as performance of crew on overall basis…

p.16 (last paragraph, left – simulators can be used to predict performance in crisis)

Simulators are good solution/later no. Raises question about hypotheses – what were they, and are they

shown to be supported by the data collected? Not so directly stated? Authors should state hypotheses explicitly

(related to 3.). In results, they should tell whether the hypotheses were supported by

considering the variables (that we can identify within those hypotheses) for which data was collected.

10 ways to look at a research paper 10 ways to look at a research paper systematically for critical analysissystematically for critical analysis 8. presentation of results: what do they really mean? Results/interpretation: ‘technical scoring system -

reliability good. Most teams technically good at handling two crisis

scenarios, but interaction wasn’t up to the mark. Level of agreement between raters was high.

10 ways to look at a research paper 10 ways to look at a research paper systematically for critical analysissystematically for critical analysis 9. conclusions drawn: are they reasonable? Although technical performance can be considered,

perfect rating scale difficult to achieve – limits of the study.

Behavioral rating system needed some improvement. Study not able to determine whether performance was

related to experience. Rating system needed more refinement, before

determining competency/board certification.

10 ways to look at a research paper 10 ways to look at a research paper systematically for critical analysissystematically for critical analysis 10. future work/research directions: any possibilities? Rating system needed more refinement, before

determining competency/board certification. Other patient safety related initiatives in research…

including predictive modeling. Now that we’ve assessed methods of evaluation of

training, can we identify models/predictive capability for determining which variables/and by how much in ‘training’ can improve/impact patient safety – where should additional resources be spent ?

Maybe more justification for the simulations is needed…

(if simulations improving learning was not yet supported)

10 ways to look at a research paper 10 ways to look at a research paper systematically for critical analysissystematically for critical analysis 1. research ideas/question -what is the purpose? 2. what background support? - literature review 3. theoretical basis for analyzing question/hypothesis? 4. applicability-practical contribution? 5. theoretical contribution? 6. appropriate methodologies for carrying out study?

(determining variables, data collection, method used to test hypotheses, validity of measures and reliability)

7. appropriate statistical analyses and assumptions? 8. presentation of results: what do they really mean? 9. conclusions drawn: are they reasonable? 10. future work/research directions: any possibilities?

RCHE presentation-Fall ‘06

• http://video.e-enterprise.purdue.edu/regenstrief/240.wmv

• Dr. Vin Sahney, Institute of Medicine & NAE

• For more information see: http://weatherhead.case.edu/executive-education/about/instructors/vinod-sahney.cfm

For next week: • Week 5: February 4 • 9. (P2) Flynn, E.A., Barker, K.N., Gibson, J.T., Pearson,

R.E., Smith, L.A., Berger, B.A., 1996. Relationships between ambient sounds and the accuracy of pharmacists’ prescription-filling performance, Human Factors, 38 (4) 614-622. (to be reviewed as a group next week - turn in initial one page summaries in class).

• 10. (Book chapter) (P3 & P4) Groopman, J., 2007. How doctors think, Houghton-Mifflin Co.: Boston. Read 2 chapters- determined in class.

• 11. (Book chapter) (P5) Kohn, L.T., Corrigan, J.M., Donaldson, M.S. (Eds.) 2000. To Err is Human: Building a Safer Health System, Ch.2 pp. 26-48 (to be reviewed together in class).

• 12. Video/plenary talk by Don Berwick of IHI.• 13. Distribute semester project description

How doctors think

• By Jerome Groopman, Houghton Mifflin, 2007 – from NYTimes Bestseller list

• Consider ch.1-9 + epilogue & Introduction

re: semester project planning• Proposal writing workshop for Graduate

Students; Jan.30, 7pm Room 214 ABCD, Stewart Center

• Online registration at: • http://www.gradschool.purdue.edu/development