Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
General Approaches of Histopathological Image
Classification using Convolutional Neural
Networks
Jerome Cheng M.D.
Disclosures
none
Outline
Background on Convolutional Neural Networksdescriptionapplicationslayers
Image classification strategiestrain from scratch transfer learning
ToolsOrangeKeras
Convolutional Neural Networks
class of deep, feed forward artificial neural networks usually composed of convolution, pooling, fully connected layers
most commonly applied in pathology for image classification and segmentation tasks
Inspired by Dr. Hubel and Dr.Wiesel’s work on the cat visual system in the 50’s and 60’s that showed a hierarchical arrangement of neurons and some neurons were excited by lines in a particular orientation in a specific location
Artificial Neural Network Example
a1
a3
a2 𝚺 Output
w1
w2
w3
Rel
u
Activation
Function
+Bia
s
a1w1+a2w2+a3w3 + bias → Relu → Output
Convolutional
Layer
Convolutional
Layer
Convolutional
Layer
Pooling Layer
Pooling Layer
Fully Connected
Layer
Fully Connected Layer
Acceleration of CNN Progress and Adaptation
Faster PCs and GPUs
Publicly available image datasets
Open source CNN deep learning frameworks
ImageNetOnline database containing more than 14 million images belonging to 20,000 categories
Large training dataset is important in deep learning
- Several pre-trained CNN models where trained on ImageNet
Accelerated the progress of CNN research
Hosted yearly competitions
Many publicly available pre-trained CNN models were trained on ImageNet
http://www.image-net.org/
ImageNet - Images
Layers
Convolutional
Pooling
Fully connected
Activation layers
Dropout layer
Regularization layers
Low level
features
High level
features
Convolutional Neural Network Layers
Training
Loss function - measure the difference between the predicted value and the expected value
Weights are adjusted in the convolution and fully connected (dense) layers to minimize the loss function
Pre-trained CNN architectures
VGG-16
VGG-19
Inception
XCeption
MobileNet
VGG-16 architecture
Simonyan, Karen, and Andrew Zisserman. "Very
deep convolutional networks for large-scale image
recognition." arXivpreprint
Applications of CNN in Pathology
Image Classification
Image Segmentation
Object localization and classification
Image Classification
Dataset Source: Levenson RM, Krupinski EA, Navarro VM, Wasserman EA. Pigeons (Columba livia) as trainable observers of
pathology and radiology breast cancer images. PLoSOne 2015;10:e0141357.
Image Segmentation
Ground Truth Prediction
Object Detection and Classification
Hung J, Carpenter A (2017) Applying Faster R-CNN for Object Detection on Malaria Images, IEEE
Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 808–813
Classification Strategies (CNN)
Train from scratch
- Weights from all layers are updated
Transfer learning
- Freeze lower layers - Only weights from some layers are updated
- Combine pre-classification layer output with machine learning algorithms
- None of the weights are updated
Transfer Learning - Freezing Layers
Weights are updated only on selected layers
Information learned from a previous dataset (ImageNet) is preserved
Speeds up training
Transfer Learning - Combine Output with Another Machine Learning Algorithm
The final classification layer of the pre-trained network should not be included
Ex. an image processed through VGG 16 = 4096 features
4096 features + image label can be trained using various machine learning algorithms like Random Forest, logistic regression, and SVM
None of the weight parameters are updated during training
Steps – Classifying Images with CNN
Scan slides
Create and label tiles - ex. 224x224 or 299x299 pixels
Training and Validation set
Augmentation
Choose or Create a Model
Model Training
Prediction
Augmentation
Rotation
Shear
Channel Shift
Scaling
Height shift
Width shift
DemsarJ, CurkT, ErjavecA, GorupC, HocevarT, MilutinovicM, MozinaM, PolajnarM, ToplakM, StaricA, StajdoharM, UmekL, ZagarL, ZbontarJ,
ZitnikM, ZupanB (2013) Orange: Data Mining Toolbox in Python. Journal of Machine Learning Research 14(Aug):2349−2353.
Chollet F. Keras [Internet] San Francisco (CA): GitHub, Inc.; 2017. [cited 2019 Apr 26]. Available from: https://github.com/keras-team/keras.
Orange (Biolab) - https://orange.biolab.si
Data preparation
Create image tiles from the digital slide or image (224 x 224 for VGG 16)
Create a folder with subfolders for each image Class
Ex. Training Set
Cancer
Benign
Orange - loading images
Orange - Image Embedding
Orange - Hardware (image embeddings)
Cluster of 8 machines
6 machines- 2 processors with 4 cores each (Intel Xenon CPU), 32 GB RAM
2 machines - 2 processors with 8 cores (Intel Xeon e5-2660), 157 GB RAM
Embedders are automatically assigned to available CPU cores
Soon: switch to an improved backend with a combination of CPUs and GPUs, with the addition of 3 TitanX GPUs (12 GB ram each)
Orange - Test and Score
Orange - data visualization
Dimensional reduction + scatter plot
Hierarchical clustering
Keras
High level neural network library that works on top of TensorFlow, CNTK, or Theano
Written in Python
CNN models can be built and trained with very little code
Works with CPU and GPUs
Summary
Convolutional Neural Networks can be used to classify and segment histopathological images
Transfer learning applies knowledge learned from a different dataset onto a new image subject matter
Orange (Biolab) is a GUI based machine learning tool that can be used for transfer learning
Keras is a high level framework written in Python for deep learning
References
Chollet, François. Deep Learning with Python. Shelter Island, New York : Manning Publications Co., 2018.
Cheng J. Convolutional neural networks. PathologyOutlines.com website. http://www.pathologyoutlines.com/topic/informaticsconvnet.html. Accessed March 25th, 2019.
Levenson RM, Krupinski EA, Navarro VM, Wasserman EA. Pigeons (Columba livia) as trainable observers of pathology and radiology breast cancer images. PLoSOne 2015;10:e0141357.
Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXivpreprint
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo,Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis,Jeffrey Ronneberger O., Fischer P., Brox T. (2015) U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab N., Hornegger J., Wells W., Frangi A. (eds) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. MICCAI 2015. Lecture Notes in Computer Science, vol 9351. Springer, ChamDean,Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow,Andrew Harp, Geoffrey Irving, Michael Isard, Rafal Jozefowicz, Yangqing Jia,Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Mike Schuster,Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Jonathon Shlens,Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker,Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas,Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke,Yuan Yu, and Xiaoqiang Zheng.TensorFlow: Large-scalemachine learning on heterogeneous systems,2015. Software available from tensorflow.org.
References
Wurtz RH. Recounting the impact of Hubel and Wiesel. J Physiol. 2009;587(Pt 12):2817–2823. doi:10.1113/jphysiol.2009.17020
HUBEL DH, WIESEL TN. Receptive fields of single neurones in the cat's striate cortex. J Physiol. 1959;148(3):574–591.
Olga Russakovsky*, Jia Deng*, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg and Li Fei-Fei. (* = equal contribution) ImageNet Large Scale Visual Recognition Challenge.
DemsarJ, CurkT, ErjavecA, GorupC, HocevarT, MilutinovicM, MozinaM, PolajnarM, ToplakM, StaricA, StajdoharM, UmekL, ZagarL, ZbontarJ, ZitnikM, ZupanB (2013) Orange: Data Mining Toolbox in Python. Journal of Machine Learning Research 14(Aug):2349−2353.
Chollet F. Keras [Internet] San Francisco (CA): GitHub, Inc.; 2017. [cited 2019 Apr 26]. Available from: https://github.com/keras-team/keras.