HMM Technology in SurgeryHMM Technology in Surgery

Timothy KowalewskiTimothy Kowalewski

BioBioRobotics LaboratoryRobotics LaboratoryBlake Hannaford, PhDBlake Hannaford, PhD

Jacob Rosen, PhDJacob Rosen, PhD

Support: National Science Foundation, ITR ProgramJohn’s Hopkins University, Greg Hager, Allison Okamura, Russell Taylor

& UW Center for Videoendoscopic Surgery, Mika Sinanan, et al.

OVERVIEWOVERVIEW

Part I:Part I:

HMM Background and Experimental SetupHMM Background and Experimental Setup

Initial Approach: Initial Approach: White Box HMM’sWhite Box HMM’s

Current Approach: Current Approach: Black Box HMM’sBlack Box HMM’s

A Step Back: A Step Back: The Bigger PictureThe Bigger Picture

Research Goals and Future WorkResearch Goals and Future Work

OVERVIEWOVERVIEW

Part II:Part II:

HMM Implementation Issues (noise, HMM Implementation Issues (noise, sufficient training, surgical ‘babbling’) sufficient training, surgical ‘babbling’)

Lessons in VQ Lessons in VQ

Toolkits availableToolkits available

Variations on a Theme: different Variations on a Theme: different approaches to surgery via HMM’sapproaches to surgery via HMM’s

Part I: HMM Background & Part I: HMM Background & Experimental SetupExperimental Setup

Surgery Surgery Signals Signals

BioRobotics and HIT Lab Surgical DatabasingBioRobotics and HIT Lab Surgical Databasing

HMM IntroHMM Intro

Computers are ‘deterministic’, so try that…Computers are ‘deterministic’, so try that…

-record a waveform and do a file-compare-record a waveform and do a file-compare

0 50 100 150 200 250-2

-1

0

1

2

0 50 100 150 200 250-2

-1

0

1

2

?==

Problem: we want to detect a ‘word’-- not Problem: we want to detect a ‘word’-- not an event or quantity …we need an event or quantity …we need AbstractionAbstraction

HMM Intro: in speechHMM Intro: in speech

Need to transcend:Need to transcend:--timing of utterances--timing of utterances--pitch, tone and volume--pitch, tone and volume

--rates of speech--rates of speech --accents--accents --noise and ‘variation,’ etc--noise and ‘variation,’ etc in order to identify the ‘words’in order to identify the ‘words’

Hopefully, adapt to errors, changes in Hopefully, adapt to errors, changes in

language, and phonetics.language, and phonetics.

…Abstraction…

HMM IntroHMM Intro

The ‘Black Box’ Approach:The ‘Black Box’ Approach:

know input expected outputTRAIN: ?

some input correct outputUSE: X

HMM Intro: BasicsHMM Intro: Basics

Markov ChainsMarkov Chains

--States --States

--Observations--Observations

State Transition Matrix (A)State Transition Matrix (A)

Initial State Distribution (Initial State Distribution ( ) )

Of the States themselves! (O)Of the States themselves! (O)

Markov ModelsMarkov Models

Markov ChainsMarkov Chains

OA ,,

SCSRRSSO

SCR

aaa

aaa

aaa

S

C

R

aaa

aaa

aaa

A

SCR

SSSCSR

CSCCCR

RSRCRR

001

8.1.1.

2.6.2.

3.3.4.

3

2

1

321

333231

322221

131211

HIDDENHIDDEN Markov Models Markov Models-- States are not observable (or even physically representable)-- States are not observable (or even physically representable) -- Observations are probabilistic functions of state-- Observations are probabilistic functions of state -- State transitions are still probabilistic-- State transitions are still probabilistic

Hungry Sleepy Asleep

Dead

Wake Up

stop breathing (.01)

Don’t start breathing(100%)

stop breathing (.01)

eat

HMM’sHMM’s

The Balls and Urns FormulationThe Balls and Urns Formulation

Observation: Color & Source Urn (seq.)

[R #3]

1 2 3

RGB RGB RGB

HMM’sHMM’s

The Balls and Urns FormulationThe Balls and Urns Formulation

1 2 3

RGB RGB RGB

Observation: Color ONLY

[R ?]

HIDDEN

HMM’sHMM’s

The Balls and Urns FormulationThe Balls and Urns Formulation

1 2 3

R-100%G- 0%B- 0%

Observation: Color ONLY

[R P(1)=1]

R- 0%G- 100%B- 0%

R- 0%G- 0%B- 100%

HMM’sHMM’s

The Balls and Urns FormulationThe Balls and Urns Formulation

Totally Distinct(not HMM)

Weighted Mix(HMM)

Uniform Distribution

HMM’sHMM’s1.1. Evaluation:Evaluation:

--of ‘fit’--of ‘fit’

2.2. Inference:Inference:

--of--of ‘hidden’ state sequence (Synthesis/ AI)‘hidden’ state sequence (Synthesis/ AI)

3.3. Training:Training:

--Capturing the Abstract--Capturing the Abstract

(4.) (4.) Comparison:Comparison:

--Another method of evaluation?--Another method of evaluation?

Infer Seq.

Oq

Evaluate Fit

OP

“Train”

OI.C.Compare

?==1 2

Surgery via SignalsSurgery via Signals

“ “It’s no longer blood and guts, it’s bits and bytes”It’s no longer blood and guts, it’s bits and bytes”

(Col. Richard Satava, MD) Prof. Of Surgery, UW; DARPA(Col. Richard Satava, MD) Prof. Of Surgery, UW; DARPA

The Blue DragonThe Blue Dragon

(Blue Signals)(Blue Signals)

Surgery via SignalsSurgery via Signals

“ “It’s no longer blood and guts, it’s bits and bytes”It’s no longer blood and guts, it’s bits and bytes”

(Col. Richard Satava, MD) Prof. Of Surgery, UW; DARPA(Col. Richard Satava, MD) Prof. Of Surgery, UW; DARPA

VR Simulator DatabasesVR Simulator Databases

Chuck Edmond MD (PI) &Lockheed Martin

Over 100 SubjectsOver 100 Subjects

400+ Total Trials400+ Total Trials

Over 100 SubjectsOver 100 Subjects

CommercializingCommercializing

Urology / TURPUrology / TURP ENT / ESSENT / ESS

Robert Sweet MD (PI) &UW HIT Lab

ESS Simulator: Tool PositionESS Simulator: Tool Position

Non-MD Junior Resident

Senior Resident Staff ENT

Blue Dragon: Tool PositionBlue Dragon: Tool Position

Expert

Novice

Left Hand Right Hand

Blue Dragon: Tool TorquesBlue Dragon: Tool Torques

Expert

Novice

Left Hand Right Hand

Part I: Initial Approach:Part I: Initial Approach:

White Box HMM’sWhite Box HMM’s

Surgery - Language ElementsSurgery - Language Elements

(Taxonomy)(Taxonomy)

White Box TrainingWhite Box Training

Force/Torque SignaturesForce/Torque Signatures

(State Mesh)(State Mesh)

(Model in P explanation)(Model in P explanation)

1 R

2 R

3 R

4 R

5 R

6 R

8 R

9 R

10 R

11 R

12 R

13 R

14 R

15 R15 L

14 L

13 L

12 L

11 L

10 L

9 L

8 L

7 L

6 L

5 L

4 L

3 L

2 L

1 L

7 R

(PIP Video)(PIP Video)

Learning Curve - Markov Model Learning Curve - Markov Model Statistical DistanceStatistical Distance

R1 R2

R3

R4

R5

E

Normalized Statistical DistanceNormalized Statistical Distance

Normalized Subjective Score Normalized Subjective Score (Video Analysis)(Video Analysis)

Normalized Completion TimeNormalized Completion Time

Normalized Trajectory LengthNormalized Trajectory Length

Correlation Between Subjective and ObjectiveCorrelation Between Subjective and ObjectiveAssessment of Surgical SkillAssessment of Surgical Skill

Part I: Current Approach:Part I: Current Approach:

Black Box HMM’s &Black Box HMM’s &

Vector QuantizationVector Quantization

White Box vs. Black BoxWhite Box vs. Black Box

Open/Defined vs. HiddenOpen/Defined vs. Hidden Based on Extant Human Knowledge vs. Based on Extant Human Knowledge vs.

Unobservable or Non-intuitive AssertionsUnobservable or Non-intuitive Assertions Procedure Specific vs. Procedure Procedure Specific vs. Procedure

independent of Cross-Proceduralindependent of Cross-Procedural Precise vs. Unpredictable Precise vs. Unpredictable

… … Grey Box ?Grey Box ?

HMM’s By AnalogyHMM’s By Analogy

Speech RecognitionSpeech Recognition

Words/Vocabulary (dictionary)

Grammar/Syntax (expression)

SurgerySurgery

Surgical ‘words’

(VQ codebook)

Surgical Syntax

(Trained HMM)

VQ – IntroVQ – Intro

Purpose: Discretization & AbstractionPurpose: Discretization & Abstraction Used Initially in Image Processing / Used Initially in Image Processing /

CompressionCompression

(VQ intro1)(VQ intro1)

(VQ intro2)(VQ intro2)

(VQ intro3)(VQ intro3)

(VQ intro4)(VQ intro4)

(VQ intro5)(VQ intro5)

(VQ intro6)(VQ intro6)

(VQ intro7)(VQ intro7)

(VQ intro 3D)(VQ intro 3D)

(VQ distortion Curve)(VQ distortion Curve)

Suturing Dictionary: VQSuturing Dictionary: VQ

Different Vocabularies Different Vocabularies across skill levels w/VQacross skill levels w/VQ

(VQ % word use)(VQ % word use)

HMM’s By Analogy (again)HMM’s By Analogy (again)

Speech RecognitionSpeech Recognition

Words/Vocabulary (dictionary)

Grammar/Syntax (expression)

SurgerySurgery

Surgical ‘words’

(VQ codebook)

Surgical Syntax

(Trained HMM)

…Abstraction…

We Want:We Want:

The ‘Black Box’ Approach:The ‘Black Box’ Approach:

know input expected outputTRAIN: ?

some input correct outputUSE: X

Authoritative Performance Authoritative Performance StandardStandard

Black Box Black Box

PerformancePerformanceSurgeon’s AnalysisSurgeon’s AnalysisWhite Box ResultsWhite Box Results

HMM ParametrsHMM ParametrsHuman KnowledgeHuman Knowledge

VQ AssumptionVQ Assumption

Variable Under TestVariable Under Test

Performance StandardPerformance Standard

Variables To Consider:Variables To Consider:

Initial Conditions (Number of Iterations)Initial Conditions (Number of Iterations) Complexity of Model: Number of StatesComplexity of Model: Number of States Amount of Human Knowledge: State Amount of Human Knowledge: State

definitions, transition eliminationsdefinitions, transition eliminations Presence of Data HybridsPresence of Data Hybrids Presence of ContextPresence of Context Direction of Analysis (more or less ?)Direction of Analysis (more or less ?)

Sample Trial Using Random Sample Trial Using Random Initializations:Initializations:

Demo of # of I.C. Demo of # of I.C. Iterations …elegance?Iterations …elegance?

Why Bother ? Why Bother ?

Can Hidden markov modeling offer Can Hidden markov modeling offer cross-cross-proceduralprocedural surgical skill assessment? surgical skill assessment?

Can the technology be Can the technology be optimizedoptimized and and embeddedembedded??

Part I: A Step Back:Part I: A Step Back:

The Bigger PictureThe Bigger Picture

Studying Tools or Studying Tools or Studying Surgery?Studying Surgery?

‘…‘…the trees were in the way’the trees were in the way’

Precision vs. AccuracyPrecision vs. Accuracy

Implementation Previous Progress vs. Lost Implementation Previous Progress vs. Lost in Analogyin Analogy

HMM’s are NOTHMM’s are NOT

A complete replacement of human surgeons A complete replacement of human surgeons or human accreditation – rather a tool for or human accreditation – rather a tool for assessment or surgeryassessment or surgery

HMM’s are not:HMM’s are not:

……the stand-alone analysis platform for the stand-alone analysis platform for surgical skill –rather a gray box, i.e., ‘both surgical skill –rather a gray box, i.e., ‘both and” not “either or” …cumulatives, errors, and” not “either or” …cumulatives, errors, etc. etc.

……without limitations.without limitations.

LimitationsLimitations

Meant to assess and develop ONLY the Meant to assess and develop ONLY the ‘basic’ surgical skill set… ‘only 300 words ‘basic’ surgical skill set… ‘only 300 words required for daily speech’ required for daily speech’

Not inNot in

Research Goals &Research Goals &Future WorkFuture Work

““The OR of the Future”The OR of the Future”

Analysis Goals:Analysis Goals:

Obtain data-rich, diverse surgical databaseObtain data-rich, diverse surgical database(BlueDragon, VR TURP, ENT Simulator)(BlueDragon, VR TURP, ENT Simulator)

Realize standardized VQ for entire databaseRealize standardized VQ for entire database

Run HMM analysis (8 toolkits so far): Run HMM analysis (8 toolkits so far): white box vs. black boxwhite box vs. black box

Find optimal HMM parameters to characterize Find optimal HMM parameters to characterize surgerysurgery

Embed the system on a single chip and it Embed the system on a single chip and it implement in hardwareimplement in hardware

Looking AheadLooking Ahead

With a surgical language developed and an With a surgical language developed and an HMM database trained, embeddable HMM database trained, embeddable possibilities include…possibilities include…

• Real-time surgical evaluation/assistanceReal-time surgical evaluation/assistance• Standardized skill set developed for surgeonsStandardized skill set developed for surgeons• Artificial Intelligence for surgical assistanceArtificial Intelligence for surgical assistance• Bandwidth reduction for tele-operationBandwidth reduction for tele-operation

Acknowledgments:Acknowledgments:

JHU: Greg Hager NSF ITR (funding), Allison Okamura, Russ TaylorJHU: Greg Hager NSF ITR (funding), Allison Okamura, Russ Taylor

BRL: Blake Hannaford, Jacob Rosen, Jeff Brown BRL: Blake Hannaford, Jacob Rosen, Jeff Brown http://brl.ee.washington.http://brl.ee.washington.eduedu

UW Med/CVES: Mika Sininan, Lily Chang, Rob Sweet, Rick SatavaUW Med/CVES: Mika Sininan, Lily Chang, Rob Sweet, Rick Satava

HIT Lab: Rob Sweet, Suzanne Weghorst, Jeff Berkely, Ganesh HIT Lab: Rob Sweet, Suzanne Weghorst, Jeff Berkely, Ganesh Sankaranaraynan Sankaranaraynan

OVERVIEWOVERVIEW

Part II:Part II:

Lessons in VQLessons in VQ

HMM Implementation Issues (noise, HMM Implementation Issues (noise, sufficient training, surgical ‘babbling’) sufficient training, surgical ‘babbling’)

Variations on a Theme: different Variations on a Theme: different approaches to surgery via HMM’sapproaches to surgery via HMM’s

Toolkits availableToolkits available

VQ – LessonsVQ – Lessons

HandednessHandedness Data-scalingData-scaling Data Type /preservation of ‘importance’Data Type /preservation of ‘importance’ Data Appending/transforming/conditioningData Appending/transforming/conditioning Different Algorithms and HMM contextDifferent Algorithms and HMM context

(display data streams)(display data streams)

(4-D Data Stream,(4-D Data Stream,GUI Plots &GUI Plots &

VQ Code-Book)VQ Code-Book)

HMM Implementation IssuesHMM Implementation Issues

The Surgical Babbling problem & NoiseThe Surgical Babbling problem & Noise O vector choice & Multiple Observations per stateO vector choice & Multiple Observations per state Continuous vs. Discrete Continuous vs. Discrete State durationState duration Choice of Data sampling, Decimation/interpolationChoice of Data sampling, Decimation/interpolation Order of ModelOrder of Model ContextContext Lots of variables (Highly Unconstrained)Lots of variables (Highly Unconstrained) Sufficient TrainingSufficient Training

Continuous vs. DiscreteContinuous vs. Discrete

Heads Or TailsHeads Or Tails

VsVs

Weight, size, etc.Weight, size, etc.

50.32

State DurationState Duration

P(State Duration) vs. Sample TimeP(State Duration) vs. Sample Time

Std HMM state duration Std HMM state duration

Variable Duration Variable Duration

Prob(in AProb(in A1111 for 100 samples) = A for 100 samples) = A1111100100

<< 1<< 1

Choice of Data sampling Rate, Choice of Data sampling Rate, Decimation /interpolationDecimation /interpolation

P(State Duration) vs. Sample TimeP(State Duration) vs. Sample Time

1 sec

Stay in one State for Stay in one State for one second …one second …

… …State duration, State duration, decimation, or decimation, or interpolation are interpolation are possible solutionspossible solutions

ContextContext

Sample Application of HMM’s (speech)Sample Application of HMM’s (speech)

--Real Life Implementation----Real Life Implementation--

--Build in some ‘determinism’… --Build in some ‘determinism’… “eyoo”“eyoo”

--Train Distinct phonetic units (phonemes) with lots of samples and --Train Distinct phonetic units (phonemes) with lots of samples and variability (same abstractive quality)variability (same abstractive quality)

‘too’ –p1

‘to’ –p2

‘two’ –p3

‘tool’ –p4

Prob. Recognition Context and Grammar Link‘too’ + (adverb)

‘to’ + (preposition)

‘two’ + (pl. noun)

‘tool’

(pronoun) +

(article / adj) +

Context for SpeechContext for Speech

Sample Application of HMM’s (speech)Sample Application of HMM’s (speech)

… …easily translates into VQ paradigmeasily translates into VQ paradigm

Block Diagram, Continuous Speech RecognitionBlock Diagram, Continuous Speech Recognition

HMM Implementation HMM Implementation ConclusionConclusion

Lots of variables (Highly Unconstrained)Lots of variables (Highly Unconstrained)

Hope for Sufficient Training …only have Hope for Sufficient Training …only have trial and error to base results on.trial and error to base results on.

Variations on HMM’s in Surgery Variations on HMM’s in Surgery

Multiple Model HMM’s: benefits (where to Multiple Model HMM’s: benefits (where to segment the tasks/models)segment the tasks/models)

Adaptive LearningAdaptive Learning # of states (automated) dynamic of States# of states (automated) dynamic of States Modeling Intention/Mental StatesModeling Intention/Mental States ““Plato, Suture Here”Plato, Suture Here”

Toolkits AvailableToolkits Available

C Code:C Code:

——HTK 3.x HTK 3.x (C-code)(C-code)

Industrial StandardIndustrial Standard

Hybrid modelsHybrid models

speech-specificspeech-specific

——UHMM UHMM (C-code)(C-code)

TestingTesting

——Past BRL Projects Past BRL Projects (C)(C)

— —JPL C-CodeJPL C-Code

— —Previous Thesis WorkPrevious Thesis Work

(contains modified HMM for(contains modified HMM for

tweaking state durations)tweaking state durations)

——GNU DistributionGNU Distribution

——Grant’s @ JHUGrant’s @ JHU

MATLAB Code:MATLAB Code:

——MATLAB Statistics 4.1MATLAB Statistics 4.1

Has VQ extras and HMM’sHas VQ extras and HMM’s

Seems stable/robustSeems stable/robust

Discrete only!Discrete only!

——UW Toolkit UW Toolkit (Matlab, BRL)(Matlab, BRL)

full spectrum of modelsfull spectrum of models

tutorial linked + GUItutorial linked + GUI

Beta-VersionBeta-Version

——H2M Toolkit H2M Toolkit (Matlab, French)(Matlab, French)

GNU DistributionGNU Distribution

limited spectrum of uselimited spectrum of use

To Do:To Do:

Get HITL simulation people to hook up w/ Get HITL simulation people to hook up w/ sutchering sim and Haptics AR Toolkit. sutchering sim and Haptics AR Toolkit. (Jeff B, Ganesh, Suzanne, new MechE guy)(Jeff B, Ganesh, Suzanne, new MechE guy)

Develop possibilities for Joint project(s).Develop possibilities for Joint project(s). Haptics-E infoHaptics-E info