IMAGERET Detection and Decision-Support Diagnosis of Diabetic Retinopathy Using Machine Vision

June 25 2009

Prof. Heikki Kälviäinen et al., ImageRet Project1

IMAGERET Detection and Decision-Support

Diagnosis of Diabetic Retinopathy Using Machine Vision

Professor Heikki Kälviäinen et al.

Machine Vision and Pattern Recognition Laboratory (MVPR)

Department of Information TechnologyFaculty of Technology Management

Lappeenranta University of Technology (LUT), [email protected]://www.lut.fi/~kalviai

http://www.it.lut.fi/ip/research/mvpr/

June 25 2009


London

Berlin

Moscow

St.Petersburg

Tallinn

Lappeenranta

Oslo

Stockholm

Helsinki

FINLAND

Lappeenranta University of Technology (LUT)

June 25 2009


Outline

• Machine Vision and Pattern Recognition Laboratory.

• Diabetes and retina. • ImageRet project and the consortium.• Objectives and results.• On-going research and future challenges.

June 25 2009


LUT Information Technology (LUT IT)

• Leader: Prof. Heikki Kälviäinen. • 6 Professors, 70 members, 850 B.Sc./M.Sc./Ph.D. students

in total, 60-70 masters and 4-5 doctors per year. • Laboratories:

– Machine Vision and Pattern Recognition (MVPR). • LUT Center of Excellence in Research.

– Software Engineering (SWE). – Communications Software (CS).

LUT IT: http://www.it.lut.fiMVPR: http://www.it.lut.fi/ip/research/mvpr/

June 25 2009


MVPR Laboratory: Research Profile

June 25 2009


Machine Vision and Pattern Recognition Laboratory (MVPR)

• Leader: Prof. Heikki Kälviäinen. • 2nd largest computer vision research group in Finland. • Center of Excellence in Research in LUT.• 24 members:

• 3 Professors + 3 Post docs + 2 Visiting doctors + 11 PhD students + undergraduate students + industry coordinator.

• Co-operation with 14 international universities and research institutes.

• Results: 18 Ph.D. degrees (and 3 externally produced), over 400 scientific publications, 40 research projects, and spin-off companies.

• Objectives: 2 PhDs/year. • Annual external project funding 700.000 EUR, basic funding

300.000 EUR, total 1.0 million EUR.http://www.it.lut.fi/ip/research/mvpr/

June 25 2009


Diabetes• Diabetes is a metabolic disorder where the blood glucose level has

been increased.• Two types:

– Type 1: insulin dependent (mostly children and young persons).– Type 2: non-insulin dependent (mostly middle-aged and elderly people).

• Diabetes is a serious disease: – When left untreated, diabetes can lead to serious medical complications

of the kidneys, the peripheral nervous system, the eyes, and it can also cause cardiovascular diseases.

• Diabetes is a common and rapidly increasing disease: – For example, 500 000 Finns (10 % of the population) are having Type 1

(in Finland the most common in the world) or Type 2 (In Finland the most common in the Nordic countries), with the increase of the number of Type 2 diabetics by 70 % in the next 10 years!

• Diabetes is an expensive disease: – 12 % of total health service costs in Finland.– Approx. 15% of all health care expenses in EU go to the treatment of

diabetes and its complications.

June 25 2009


Diabetic Retinopathy• Diabetic retinopathy is a disease of the retina (the tissue

responsible for vision in the eye) caused by diabetes.• Without proper treatment it can lead to the loss of vision or

even blindness (the leading cause of the blindness in the working age population).

• Early detection of the retinal complications is crucial.• An ophthalmic fundus camera can be used to monitor the

condition of the retina => fundus photoghaphy.

June 25 2009


Fundus Image AcquisitionThe eye: Besides the vision system, an useful peephole inside a human

being to see what is happening. Zeiss Fundus Camera:1500 x 1152 pixels 24 bits per pixel.

June 25 2009


June 25 2009


Challenges and Objectives• Challenges: robust screening needed.

• 500.000 diabetes patients in Finland and the number is increasing.

• How to monitor the known patients and find the new ones?

• Not enough medical experts nor funding for applying current practices. => We must find robust automatic or semiautomatic solutions for two tasks: 1. To decide whether the eye is healthy or not (the disease

present or not).

2. To find reliably abnormalities in the eye, if the eye is considered to be not healthy.

June 25 2009


ImageRet Project• FinnWell technology program was established by Finnish National

Agency for Technology and Innovation (TEKES).• The project called Optimal Detection and Decision-support Diagnosis

of Diabetic Retinopathy (ImageRet) was established to develop reliable and accurate image processing and pattern recognition methods for automatic fundus analysis.

• Project of 38 months in 2006-2009 with the several partners (750.945 EUR):• Lappeenranta University of Technology (LUT), Finland: project coordination,

machine vision and pattern recognition.• University of Kuopio, Finland (UKU): ophthalmology.• University of Joensuu, Finland (UJO): spectral imaging. • Mikkeli Polytechnics, Finland (MAMK): databases and metadata. • University of Bristol, UK (UB): optic disk detection. • Companies: Kuomed Oy, Mawell Oy ,Perimetria Oy, Pfizer Oy, Santen Oy,

VAS Oy.

http://www.it.lut.fi/project/imageret/

June 25 2009


ImageRet: Acknowledgements Intensive co-operation of many researchers

Pauli Fält (UJO), Jari Forström (MAMK), Pertti Harju (MAMK), Dr. Jouni Hiltunen (UJO), Dr. Markku Hauta-Kasari (UJO), Valentina Kalesnykiene (UKU), Tomi Kauppi (LUT), Markku Kuivalainen (LUT), Prof. Joni Kämäräinen (LUT), Prof. Heikki Kälviäinen (LUT), Dr. Lasse Lensu (LUT), Mika Letonsaari (MAMK), Prof. Majid Mirmehdi (UB), Pekka Nikula (LUT), Prof. Jussi Parkkinen (UJO), Dr. Juhani Pietilä (Perimetria), Markku Rossi (MAMK), Prof. Iiris Sorri (UKU), Prof. Hannu Uusitalo (UKU), etc.

& many company representatives.

June 25 2009


ImageRet: Objectives

• Image annotation tool for medical expert annotation. – Medical experts can save and compare their diagnoses with the tool.

• Fundus image databases. – Expert annotations collected as ground truth in public databases. – Testing protocols for benchmarking between different methods. – Private patient databases (including temporal changes in the eye).

• Evaluation framework.– A solid basis for the image analysis system development and

comparison. • Image-based and pixel-based methods.

– Image-based: Is there a healthy eye or not in an image? – Pixel-based: Detection of abnormalities related to diabetic retinopathy:

hard exudate, soft exudate, hemorrhage, microaneurysm, and neovascularization.

• Spectral imaging. • How much more it can be “seen” using spectral imaging?

June 25 2009


Normal Fundus Image

1. Papilla (optic disk).

2. Blood vessels.3. Macula (the area

of the sharp vision).

1.

2.

3.

June 25 2009


Hard Exudate• Hard exudate

consists of blood plasma and lipids leaked from blood vessels.

• It is one of the most commonly occurring lesion in diabetic retinopathy.

• Yellow-white lesions.

• Sharp margins.

June 25 2009


Soft Exudate

• Soft exudate is a micro-infarct occurring in an eye.

• Yellowish lesions.

• Fuzzy margins.

June 25 2009


Hemorrhage

• Hemorrhage consists of blood leaked from vessels.

• Dark red lesions.• The color is

quite similar as in vessels.

June 25 2009


Microaneurysm

• Out-pouching of capillary.

• Visible as a tiny red dot.

• The first observable type of lesion in retinopathy.

• Quite difficult to notice in a color fundus image.

June 25 2009


Neovascularization

• Abnormal vessels growing to satisfy the lack of oxygen in a retinopathic eye.

• Can cause severe problems.

June 25 2009


Medical Expert Annotations

Digital fundus image. Medical expert annotations.

June 25 2009


Image Annotation Tool

June 25 2009


Fundus Image Databases

Databases • DIARETDB0• DIARETDB1• DIARETDB1 V2.1

publicly available at

http://www.it.lut.fi/project/imageret/

June 25 2009


Diabetic Retinopathy Database

Images (89):Train images 28Test Images 61Med. experts 4Findings:Haemorrhages (Ha)Microaneurysms (Ma)Hard exudates (He)Soft exudates (Se)

June 25 2009


Evaluation Framework

June 25 2009


Evaluation Framework – Training

• Uneven illumination. • Colour distortions.

Imaging related. Eye related.

• Imaging noise.

• Colour.• Texture.• Shape.

June 25 2009


Fusing Multiple Medical Expert Annotations

(b)(a) (c)

Annotation fusion approaches*: a) weighted area intersection b) representative point neighbourhood c) representative point neighbourhood masked.

* Tomi Kauppi, Joni-Kristian Kämäräinen, Lasse Lensu, Valentina Kalesnykiene, Iiris Sorri, Heikki Kälviäinen, Hannu Uusitalo, Juhani Pietilä, Proc. of the 16th Scandinavian Conference on Image (SCIA2009), Fusion of multiple expert annotations and overall score selection for medical diagnosis.

June 25 2009


Feature Extraction - Colour as Feature

Diabetic retinopathy colour distributions

June 25 2009


Estimating Colour Distributions: Learning

Estimating colour distributions with a Gaussian mixture model using the unsupervised Figueiredo-Jain algorithm.

June 25 2009


Evaluation Framework – Analysis

June 25 2009


Analysis – Classification (Colour) 1/2

Pixel-wise likelihood for hard exudates: a) original image;

b) probability density map (likelihood) for colour (RGB).

•a) •b)

June 25 2009


Analysis – Classification 2/2

• The pixel-wise probability of diabetic finding, p(finding), for image is the combination of the selected probability density maps:

p(finding) = p('colour')p('texture')p('reliability' )...

June 25 2009


Analysis – Overall Score Fusion

• Test hypothesis = the disease present (positive) or not (negative).

• Overall score = a test outcome indicator for an image (a higher value increases the certainty of the positive outcome).

• Overall score fusion strategies*: max, summax, mean, product.

* Tomi Kauppi, Joni-Kristian Kämäräinen, Lasse Lensu, Valentina Kalesnykiene, Iiris Sorri, Heikki Kälviäinen, Hannu Uusitalo, Juhani Pietilä, Proc. of the 16th Scandinavian Conference on Image (SCIA2009), Fusion of multiple expert annotations and overall score selection for medical diagnosis.

June 25 2009


Evaluation Framework – Evaluation

Sensitivity. Specificity.

* Tomi Kauppi, Valentina Kalesnykiene, Joni-Kristian Kämäräinen, Lasse Lensu, Iiris Sorri, Asta Raninen, Raija Voutilainen, Hannu Uusitalo, Heikki Kälviäinen, Juhani Pietilä, Proc. of the British Machine Vision Conference (BMVC2007), The DIARETDB1 diabetic retinopathy database and evaluation protocol, pp. 252-261, Vol. 1.

June 25 2009


Evaluation – Receiver Operating Curve (ROC) Parameters

Sensitivity = T_P/(T_P+F_N)

Specificity = T_N/(T_N+F_P)

T_P = true positives (abnormal) T_N = true negatives (normal)F_P = false positives (normal as abnormal) F_N = false negatives (abnormal as normal)

June 25 2009


Evaluation – ROC Curves

= Equal error rate (EER)

= Weighted error rate (WER)

June 25 2009


Method Development So Far

• Automatic image analysis in human supervision is possible.

• Possible applications.

– Medical diagnosis assistance – screening.

– Fundus image sorting according to severity/certainty of the disease.

– Semi-automatic tool to aid remote diagnosis.

– Quality control of diagnosis work.

– Patient specific image databases.

June 25 2009


Spectral Imaging: Significantly New Information about Diabetic

Retinopathy ?• Grayscale images: 1 channel (e.g. fluorescein angiography).• RGB images: 3 channels (colour photographs).• Spectral images: Tens or hundreds of separate colour channels.

=> Contain significantly more colour information than RGB images.

Spatial

Spectrum

Spectral image

SpatialR/G/B

RGB

Spatial

Grayscale

June 25 2009

Prof. Heikki Kälviäinen et al., ImageRet Project

Article in the 16th Scandinavian Conference on Image Analysis (SCIA 2009), Oslo, Norway, June

15-19, 2009:

Extending Diabetic Retinopathy Imaging from Color to Spectra

Pauli Fält1, Jouni Hiltunen1, Markku Hauta-Kasari1, Iiris Sorri2, Valentina Kalesnykiene2, and Hannu Uusitalo2,3

1InFotonics Center Joensuu, University of Joensuu, Joensuu, Finland2Department of Ophthalmology, Kuopio University Hospital and University of

Kuopio, Kuopio, Finland3Department of Ophthalmology, Tampere University Hospital, Tampere, Finland

June 25 2009


• Built by Color Vision Group, University of Joensuu, Finland based on Canon CR-45NM fundus camera.

• Spectral separation by 30 narrow bandpass interference filters.

• 400 – 700 nm at approx. 10 nm steps.

• A digital grayscale image for each filter separately => spectral image.

Spectral Fundus Camera

June 25 2009


Human Subjects

• 66 volunteers: 54 diabetic patients + 12 control subjects.

• The clinical trials were conducted in the Department of Ophthalmology of the Kuopio University Hospital, Finland.

• A corresponding spectral database will be published soon.

June 25 2009


RGB 580 / 540 / 500 nm

Optimal Colour Channels

June 25 2009


RGB 580 / 540 / 500 nm

Can We “See” More?

June 25 2009


Yes?

June 25 2009


Summary of Results and Future Challenges• Image annotation tool for medical expert annotation.

– Done. • Fundus image databases.

– Done as defined in the objectives. – More expert annotations to verify ground truth and a new release, if

needed. – First patient databases as a function of time collected.– Spectral database to be published.

• Evaluation framework.– Done.

• Image-based and pixel-based methods. – “Semiautomatic” solution done (image-based screening). – More method development needed: spatial prior information, texture analysis,

shape analysis, spectral colour information. – Other diseases than diabetes.

• Spectral imaging. • Images taken and preliminary expert annotations marked (more needed). • Feature selection to be studied and related methods to be developed.

Documents

IMAGERET Detection and Decision-Support Diagnosis of Diabetic Retinopathy Using Machine Vision