View
4.413
Download
0
Category
Preview:
Citation preview
360|iDev 2017
Shuichi Tsutsumi @shu223
Deep Learning on iOS
Overview• How to implement “Deep Learning” on iOS
Metal Performance Shaders (MPSCNN) Accelerate (BNNS)
Core ML
Vision
Your App
Why so exciting? !
AIphaGo
Cancer Detection
Self-driving Car
AutoDraw
Pose Estimation
http://qiita.com/nnn112358/items/a4490d85dac5827db53b
Frontal View Synthesis
AutoHair
AutoDraw Pose Estimation
Frontal View AutoHair
AutoDraw Pose Estimation
Frontal View AutoHair
This evolutional tech works on iOS"
Deep Learning ON iOS
= works on iOS devices
(Demo)
Image
Result60 times / sec
xUsers
#
#
TOOTHPASTE
!
Train InferenceTrained Params
iOS ML frameworks
Implementation
"""Core ML"""
The largest hall was 100% full for the Core ML session.
✋
Metal Performance Shaders (MPSCNN) Accelerate (BNNS)
Core ML
Vision
Your App
iOS 11
iOS 10
Metal Performance Shaders (MPSCNN) Accelerate (BNNS)
Core ML
Vision
Your App
iOS 11
iOS 10
Your App
Metal Performance Shaders (MPSCNN) Accelerate (BNNS) iOS 10
• Optimized for GPU (by ) and CPU (by Accelerate) • Available for iOS 10, too • Basically any ML tools can be used to train the models
Still works!
Metal Performance Shaders (MPSCNN) Accelerate (BNNS)
Your App
GPU CPU
MPSCNN 900 results
Core ML 160,000 results
3 steps to implement w/ MPSCNN
How to implement w/ MPSCNN
Step 1: Create the model
Train InferenceTrained Params
-Which tools can be used for the training? -What kind of formats can be used to pass the pre-trained params?
Which ML tools can be used? : ANY What model format can be used? : ANY
Model (Trained Params)
dat
Train
ML Tools
Which ML tools can be used? : ANY What model format can be used? : ANY
Model (Trained Params)
dat
Train
ML Tools
hdf5
•The “.dat” files are common binary files. - Not specific for iOS or MPS.
•Contains the trained params - Weights / Biases
•Any other format can be used as long as it can be read by the iOS app. - c.f. hdf5
Which ML tools can be used? : ANY What model format can be used? : ANY
Model (Trained Params)
dat
Train
ML Tools
hdf5
•Any tools which can train CNN, and export the params
How to implement w/ MPSCNN
Step 2: Implement the network
MPSCNNConvolution
MPSCNNFullyConnected
MPSCNNPooling
MPSCNNConvolution
MPSCNNConvolution
MPSCNNPooling
MPSCNNConvolution
MPSCNNFullyConnected
MPSCNNPooling
MPSCNNConvolution
MPSCNNConvolution
MPSCNNPooling
• Almost same name -> Easy to find• Complicated Maths or GPU optimization are encapsulated
Classes corresponding to each CNN layers are provided
How to implement w/ MPSCNN
Step 3: Implement the inference
Input image
MPSImage
Result (UInt, etc.)
CNN • Implemented in Step 2 • Trained params (created in Step 1)
are loaded.
Do something
• Step 1: Create the model - Any ML tools can be used
- Any file format can be used for the trained params
• Step 2: Implement the network - Classes corresponding to each CNN layer are provided
• Step 3: Implement the inference - Input to the CNN, and output from the CNN
Demo : Logo detection
Increased by 70 times
Trained Params
dat
f.write(session.run(w_conv1_p).tobytes())
(Demo)
⚠
CNN for the logo detection
GoogLeNet (Inception v3)
GoogLeNet (Inception v3) Apple’s implementation w/ MPSCNN
Inception3Net.swift
2,000 lines&Only for the CNN
Core ML
Development Flow w/ MPSCNNML Tools
Train
some format
Trained Params
dat
Extract
Parse
dat
MPSCNNConvolution
MPSCNNFullyConnected
Implement Network 2,000
lines&App
Development Flow w/ Core MLML Tools
Train
some format
Trained Params
dat
Extract
Parse
dat
MPSCNNConvolution
MPSCNNFullyConnected
Implement NetworkApp 2,000
lines&
1) Convert w/ coremltools
2) Drag & Drop
some format
Generate
Input image
MPSImage
Result (UInt, etc.)
CNN
Do something
let size = MTLSize(width: inputWidth, height: inputHeight, depth: 1) let region = MTLRegion(origin: MTLOrigin(x: 0, y: 0, z: 0), size: size) network.srcImage.texture.replace( region: region, mipmapLevel: 0, slice: 0, withBytes: context.data!, bytesPerRow: inputWidth, bytesPerImage: 0)
Need to know Metal to use MPSCNN
let origin = MTLOrigin(x: 0, y: 0, z: 0) let size = MTLSize(width: 1, height: 1, depth: 1) finalLayer.texture.getBytes(&(result_half_array[4*i]), bytesPerRow: MemoryLayout<UInt16>.size*1*4, bytesPerImage: MemoryLayout<UInt16>.size*1*1*4, from: MTLRegion(origin: origin, size: size), mipmapLevel: 0, slice: i)
MPSCNN Accelerate (BNNS)
Core ML
Vision
Your App
Input image
MPSImage
Results
CNN
Do something
let ciImage = CIImage(cvPixelBuffer: imageBuffer)let handler = VNImageRequestHandler(ciImage: ciImage)try! handler.perform([self.coremlRequest])
Don’t need to touch Metal to use Vision
guard let results = request.results as? [VNClassificationObservation] else { return }guard let best = results.first?.identifier else { return }
MPSCNN Accelerate (BNNS)
Your App
MPSCNN Accelerate (BNNS)
Core ML
Vision
Your App
MPSCNN Accelerate (BNNS)
Core ML
Vision
Your App
How about BNNS?
Deep Learning on CPU?!
CPU: 40 days GPU: 6 days
Deep Learning on CPU?!
Asked @ Lab in WWDC17
“How can we utilize both MPSCNN and BNNS?”
' “Basically use MPSCNN.”
“OK, but, when should I choose BNNS?”
' “I don’t know.”
• He added “watchOS might be a case on that you should use BNNS.”
• Because watchOS doesn’t support MPSCNN, but supports BNNS. (I haven’t tried yet.)
My current understanding:• The cost for passing data between CPU and GPU is
not small • When the network is small, the CPU <-> GPU cost
might be bigger than the benefit of parallel processing.
BNNS might be better when the network is small.
Recap• Why is “Deep Learning on iOS” exciting? • How to implement “Deep Learning” on iOS
- w/ MPSCNN (iOS 10)
- w/ Core ML & Vision (iOS 11)
- When to choose BNNS
MPSCNN BNNS
Core ML
Vision
Thank you!
https://github.com/shu223
Recommended