HiPerNav – High Performance soft-tissueNavigation

SC meeting iSMIT 30.march 2019Heilbrunn

Ole Jakob Elle, PhDHead of Medical Cybernetics and Image Processing and Coordinator of HiPerNav,

The Interventional Centre, Rikshospitalet, Oslo University Hospital

Professor atDepartment of Informatics

University of Oslo

HiPerNav• Project Coordination: Oslo University Hospital• 13 European universities, hospitals and industrial companies

participates• 16 ESR’s : 14 full PhD´s financed and 2 partly funded through

the project• Aims to improve important bottlenecks in soft-tissue navigation:

– effective pre-operative model and planning– accurate and fast intra-operative model update – accurate and fast model-to-patient registration – Intuitive user-interaction and effective workflow– high performance computing by use of GPU

HiPerNav - Partners• Norway:

– Oslo University Hospital-IVS (Coordinator)– NTNU– Sintef– Innovation Norway (Partner organisation)

• France:– INRIA– University Paris13

• Switserland:– University of Bern– University Hospital Bern (Inselspital), (Partner organisation)– Cascination

• Spain:– University of Cordoba

• The Netherlands:– University of Delft– Yes!Delft (Partner organisation)

• Germany– SIEMENS (Partner organisation)

• United States– NVIDIA

Overview of the progress in HiPerNav

The work carried out during the reporting period is in line with the Annex 1 in the Grant Agreement, and is for this first reporting period mainly divided into the following areas: • Preparation and finalizing of the Consortium Agreement (CA) as well as

separate agreements with the Partner Organizations (WP10). • Recruitment of all the 16 ESRs in the project (WP10) • Planning and preparation for the training and research of the recruited

ESRs (WP1) • Five Training Events and School finalised: one week in Oslo, one week in

Trondheim, one week in Bern one week HiPerNav School in Cordoba and one week in Strasbourg (WP1)

• 21/21 Deliverables submitted in time

• Each ESR follow their individual training following the PhD program of the institution that they are enrolled in.

• HiPerNav Training events and Schools– First HiPerNav training event was held in Oslo, Norway in week 36, 2017 (M11) – Second HiPerNav training event was held in Trondheim, Norway in week 39, 2017 (M11). – Third HiPerNav training event was held in Bern, Switzerland in week 11, 2018 (M17). – The first HiPerNav School combined with the fourth HiPerNav training event was held in

Cordoba in week 37/38 (M23).– The fifth HiPerNav training was organized in Paris and Strasbourg in the 2nd week of

December 2018

• Secondments: – ESR1 – From OUS to SINTEF (1M).– ESR4 – From UCO to OUS (2,5M).– ESR4 – From UCO to NTNU-Gjøvik (0,5M).– ESR5 – From UCO to NTNU-Trondheim (3M).– ESR15 – From TUD to OUS (1M).– ESR14 – From SINTEF to UCO (Starting from now)

• Dissemination: 2 Journal papers ; 12 conference/workshop papers; 1 book chapter

Communication and data sharing- Deployment of web services and data release -

• HiPerNav has currently established software infrastructure/data management platform to support the dissemination, communication and data-sharing activities.

• Website (http://www.hipernav.eu) • Data-management platform (https://data.hipernav.eu)• Current availability of data

– 92 medical images: 23 subjects x (2CT + 2 MR)– Availability on request to participants signing a data-sharing agreement

Exploitation- Preparation work for exploitation in project period 2 -

• Planning of the exploitation will be performed in next period of the project.

The Cas-One Navigation system from Cascinationin a laparoscopic liver resection at The Intervention Centre, Oslo University Hospital.

ESR1 (OUS) – Teatini Andrea

Main Progress Milestones

• Implementation of workflow to reconstruct and register liver surface point clouds based on stereo-laparoscopy.

• Implementation and testing of N-posed hand-eye camera calibration to track the camera pose through optical markers.

• Completed Pilot Animal study to understand influence of liver deformation due to pneumoperitoneum on point based registration and to obtain data of optically tracked stereo-video.

• Working on automatic segmentation of Liver surface from laparoscopic video through Deep Learning to improve accuracy of AR and stereo reconstruction.

Topic: Image-to-Patient Registration

ESR2 (OUS) – Pravda Jith Ray

Main Progress Milestones

• Successful implementation of image segmentation for Liver parenchyma in CT images using Deep Learning network

• Towards completion on research to find different combinations of learning parameters that can influence ConvolutionalNeural Networks (CNNs) in accuracy of segmentations in CT images

• Working on automatic segmentation of Liver from pre-operative MR images using deep neural networks ( on 135 and 600 datasets)

Topic: Image segmentation for creating patient-specific models

ESR3 (OUS) – Egidijus Pelanis

Main Progress Milestones• Experience with laparoscopic parenchyma-sparing liver

resection for colorectal metastases. Article describing and overviewing performed procedures.

• Acquiring approvals for the use of MRI/CT and video data from Oslo CoMet study in the project. Ground truth creation.

• Receiving approval and start of animal trial to gather data and to test multiple techniques for segmentation and registration.

• Exploration of 3D liver model visualisation in mixed reality for surgery planning and guidance during surgery. Validation study comparing conventional method with the use of Microsoft HoloLens.

• Development of a liver phantom with 3D printed parts for validation of navigation system and surgical training.

Project Title: Clinical assessment and validation

ESR14 (SINTEF) – Javier Pérez de Frutos

Main Progress Milestones

• Study of the state-of-the-art of intraoperative image-to-image registration techniques.

• Testing and validation of novel Single Landmark Registration method for initial registration. (CARS 2018, Berlin)

• Collaborative study on performance of tracking systems in an operating room setup. (EMBC 2018, Honolulu)

• Collaborating on the development of a multi-modal registration algorithm of liver vascular system using Convolutional Neural Networks architectures.

Topic: Intraoperative Image-to-Image Registration

-100

-80

-80

-60

-100

-40

-120

-20

Sampling region 4

-140 100

0

Sampling region 2

-160

20

Sampling region 3

50

Vascular graph

40

-180

60

Sampling region 5

0-200

80

-50-220

Sampling region 1

-240 -100-260 -150

n865

16

-30 n865

-205

n866

-28

n867

n866

18

-26 -210

n868

n867

20

-24

Sampled subgraph 1

-215

n868

-22

n877

22

n876

n876

n878

-20

n885

-220

n877

24

n879

n878 n885

n879

26

Sampled subgraphTransformed subgraph

19-203-30

20

21

-28 -204

22

-26

Bit map representation

-205

23

-24

24

-206

25

-22

26

-207-20

19

20

21

22

23

24

25

26

27

-30

n865

n866 n867

Vascular graph

-25

n868

n877

n879

n876 n878

-203-204

n885

-205-20 -206

ESR15 (TUD) – Gholinejad Maryam

Topic: User interaction and workflow analysis in Minimally Invasive Liver Surgery

Main Progress

Ø Detailed generalized workflow of current laparoscopic liver resection surgery established through observations, expert interviews and literature reviews.

Ø Workflow analysis programs developed: HiPerNav Video Marker and HiPerNav Simulation Assistant. Used to acquire, validate and analyze workflow verification data.

Ø Qualitatively and quantitatively verified the workflow with aforementioned software.

Ø Detailed discrete events simulation model of the workflow built in Matlab. Model fed by Video Marker output data.

T _______________________The Intervention Centre