Using CoreML to Push the Limits of your App

Core ML and your AppHow to use Core ML to push the limits

Mugunth Kumar, Itty Bitty Apps, Melbourne

Agenda

This talk explains how aspiring app developers/companies can take full advantage of Core ML to push their apps to the next level.


Agenda


I'll introduce you to - The fundamentals of Core ML - What is possible with machine learning in iOS 11 - What is not possible with machine learning in iOS 11


Agenda


I'll introduce you to - The fundamentals of Core ML - What is possible with machine learning in iOS 11 - What is not possible with machine learning in iOS 11

Lastly, What could potentially be possible in a future release of iOS (12 and above

!

)


Agenda


Agenda

» What is Core ML


Agenda

» What is Core ML

» Machine Learning in iOS 11

» Vision

» Natural Language Processing


Agenda

» What is Core ML


» Vision


» Features not yet available

» Retraining models (iOS 12

!

)


Agenda

» What is Core ML


» Vision


» Features not yet available

» Retraining models (iOS 12

!

)

» Questions


Core ML

What is Core ML


Core ML

What is Core ML

» Core ML is a framework newly introduced in iOS 11 SDK


Core ML

What is Core ML


» Once considered complicated


Core ML

What is Core ML



» Once considered CPU intensive


Core ML

What is Core ML




» Once considered impossible to be done on the client


Core ML

What is Core ML




» Once considered impossible to be done on the client

» Made easy in iOS 11

!


Features available today in iOS 11



» Vision

» Object Tagging and Recognition



» Vision

» Object Tagging and Recognition


» Sentiment Analysis

» Language Tagging

» Parts of Speech Analysis


This talk

This talk is exclusively on NLP and text processing for two reasons.


This talk


» Plenty of libraries and blogs already exist on vision and Object Tagging.


This talk



» Vision tagging model files are large (30MB to 200+ MB) and the "side" benefit you get using Core ML gets defeated.


This talk



» Vision tagging model files are large (30MB to 200+ MB) and the "side" benefit you get using Core ML gets defeated.

» As of iOS 11, you can't on the fly download a Core ML model on the fly like on-demand resource.


This talk

With Core ML and NLP, your app could do


This talk




This talk



» Parts of Speech Analysis/Language Tagging


This talk



» Parts of Speech Analysis/Language Tagging

» Combining Both


Who can be benefitted by Sentiment Analysis



» Apps that show reviews




» Social media account classifiers




» Social media account classifiers

» Pretty much any app that deals with lots of text from e-book readers to magazine apps


Getting Started

You need a Core ML Model. You can download from

Or you can convert using coremltools


Getting Started


» https://developer.apple.com/machine-learning/



Getting Started



» https://github.com/likedan/Awesome-CoreML-Models



Getting Started



» https://github.com/likedan/Awesome-CoreML-Models


» https://pypi.python.org/pypi/coremltools


Sentiment Analysis Demo



» For the demo, I'll be using one such model from the Awesome-CoreML-Models repo.




» Specifically SentimentPolarity.mlmodel by Vadym Markov.




» Specifically SentimentPolarity.mlmodel by Vadym Markov.

» It's a very light weight model weighing in at 275Kb.




The Code!

func predictSentiment(from text: String) -> Sentiment { do { var inputFeatures = features(from: text) guard inputFeatures.count > 1 else { throw Error.featuresMissing }

let output = try model.prediction(input: inputFeatures)

let positiveProb = output.classProbability["Pos"]! let negativeProb = output.classProbability["Neg"]!

return Sentiment(positiveProbablity: positiveProb, negativeProbablity: negativeProb) } catch { preconditionFailure("Error occurred. Model file might be corrupted.") } }


More code!

func features(from text: String) -> [String: Double] { var wordCounts = [String: Double]()

tagger.string = text let range = NSRange(location: 0, length: text.utf16.count)

tagger.enumerateTags(in: range, scheme: .nameType, options: options) { _, tokenRange, _, _ in let token = (text as NSString).substring(with: tokenRange).lowercased()

guard token.count >= 2 else { return }

if let value = wordCounts[token] { wordCounts[token] = value + 1.0 } else { wordCounts[token] = 1.0 } }

return wordCounts }


Performance


Performance

» Reasonably fast.


Performance

» Reasonably fast.

» With more text, performance can take a hit


Performance

» Reasonably fast.

» With more text, performance can take a hit

» As long as you avoid the main thread, you should be good to go


Parts of Speech Analysis





» Parts of speech analysis (and Language Tagging) was introduced since iOS 5.




» All those calendar apps you saw were using this feature to convert your natural language input into a calendar or a reminder event.





» You don't need a model file to do this. Everything you need is built into the SDK.





» You don't need a model file to do this. Everything you need is built into the SDK.

» So why is this section in this talk?





» In iOS 11, Parts of Speech Analysis was completely reimplemented on top of Core ML framework




» Language Tagging wasn't very accurate with some Eastern European and Asian languages.




» Language Tagging wasn't very accurate with some Eastern European and Asian languages.

» This is mostly fixed with Core ML in iOS 11


Lets get our hands dirty!

func tags(from text: String) -> [[String: String]] { var tagsToReturn = [[String: String]]()

tagger.string = text let range = NSRange(location: 0, length: text.utf16.count)

// Tokenize and count the sentence tagger.enumerateTags(in: range, scheme: .lexicalClass, options: options) { tag, tokenRange, _, _ in let token = (text as NSString).substring(with: tokenRange).lowercased() let tag = (tag?.rawValue ?? "") tagsToReturn.append([token : tag]) } return tagsToReturn }


Parts of Speech Analysis Demo


Mixing them all up

There are plenty of research articles that argues sentiment analysis is better done by emphasising adverbs and adjectives.

Words like beautiful, useless, hopeless, gigantic convey more "sentiment" information compared to objects in a sentence like "Australia", "iPhone" or subjects like "I", "You".

Ex: I can clearly say, Australia is one of the most beautiful countries on the planet.


Lets get our hands dirty!

func featuresUsingTaggingAlgorithm(from text: String) -> [String: Double] {

let parsedTags = tags(from: text) var wordCounts = [String: Double]()

for tag in parsedTags { let (token, tag) = (tag.keys.first!, tag.values.first!)

var valueToAdd = Double(1.0)

if tag == "Adverb" { valueToAdd = Double(30.0) } if tag == "Adjective" { valueToAdd = Double(35.0) }

if let value = wordCounts[token] { wordCounts[token] = value + valueToAdd } else { wordCounts[token] = valueToAdd }

} return wordCounts }


MIX UP ALGORITHM DEMO


Features not yet available



» Retraining models is not possible (iOS 12 !

)

» Face Recognition can't be performed as of now

» Face Recognition first involves detection of faces (available since iOS 6)

» Pass these as training data to a classifier

» Face Recognition is however available for the native Photos app. So it is only a matter of time.



SARCASM DETECTION!


What you should know by now


What you should know by now» Apple has very elegantly abstracted out the inner

workings that, most of Core ML is fairly straight forward (codewise: under 10 lines) if you know the foundations.




» Understanding the foundations will let you apply Core ML in your app easily.




» Understanding the foundations will let you apply Core ML in your app easily.

» Using CoreML innovatively will help make your app stand out from the crowd.


That's ME!

I'm a


That's ME!

I'm a

» Developer


That's ME!

I'm a

» Developer

» Author of iOS Programming Pushing the Limits


That's ME!

I'm a

» Developer

» Author of iOS Programming Pushing the Limits

» Currently baking some cool apps @ Itty Bitty Apps, Melbourne


Questions?Mugunth Kumar, Itty Bitty Apps, Melbourne

Software

Using CoreML to Push the Limits of your App