Embed Size (px)
Citation preview
DEEP LEARNING QUALITY ASSURANCE
Marius Reuther, Founder DLQA
SOCIAL MEDIA AND AI
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 2
https://www.chefblogger.me/2016/02/18/sind-frauen-
wirklich-so-auf-social-media/
https://techcrunch.com/wp-
content/uploads/2016/03/tay.jpg?w=1390&crop=1
AI FAILING
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 3
https://techcrunch.com/wp-content/uploads/2016/03/tay.jpg?w=1390&crop=1
Tay.ai
• Microsoft‘s AI Chatbot
• Published in March 2016 on
• Emits salacious and abusive
tweets
• Shutdown only 16 hours after the
releasehttps://techcrunch.com/wp-
content/uploads/2016/03/tay.jpg?w=1390&crop=1
Semantic Segmentation Road ModelObject Detection
CAMERA AI APPLICATIONS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 4
https://www.youtube.com/watch?v=_kxX09i4fds https://www.youtube.com/watch?v=PNzQ4PNZSzc https://www.youtube.com/watch?v=ETT34fn7OfE
Collision PreventionObject Detection Semantic Segmentation
LIDAR AI APPLICATIONS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 5
https://www.youtube.com/watch?v=KdrHLXpYYlg https://www.youtube.com/watch?v=7NNpvtdrHkU https://www.youtube.com/watch?v=nXlqv_k4P8Q
AI TrainingDataset
DATASET CREATION PROCESS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 6
Oil Rig Refinery
Collect DataAI Developer
Data
Labeling
Datasets for Machine Learning in AD typically consist of 10k – 10 mio images.
Annotation may take 1h or more.
LABELING LANDSCAPE
Labeling Service Labeling ToolingTooling + Service
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 7
Computer Vision Annotation Tool
(CVAT)
LabelMe
LABELME – OPEN SOURCE LABELING TOOL
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 8
LabelMe
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 9
Car
car,
CAR,
Automobile,
Object,
Auto,
Vehicle,
Vehcl,
MotorizedVehicle,
motorized vehicle,
Motorized Vehicle,
…
LABELME – OPEN SOURCE LABELING TOOL
LabelMe
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 10
Car
car,
CAR,
Automobile,
Object,
Auto,
Vehicle,
Vehcl,
MotorizedVehicle,
motorized vehicle,
Motorized Vehicle,
…
https://www.pinterest.it/pin/423479171186745584/
LABELME – OPEN SOURCE LABELING TOOL
CVAT – OPEN SOURCE LABELING TOOLS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 11
CVAT
Matlab Ground Truth Labeler Alp‘s Labeling Tool (ALT)
MORE OPEN SOURCE LABELING TOOLS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12
Overview about labeling tools:
https://de.slideshare.net/yuhuang/annotation-102542284
OPEN SOURCE DATASETS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 13
Overview about open-source datasets:
https://de.slideshare.net/yuhuang/public-dataset-for-autonomous-driving
TIME SPEND ON CREATING AI PROJECTS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 14
https://www.scribd.com/document/381514649/TRAIN-AI-2018-Andrej-Karpathy-Tesla(
datasets
models and algorithms
GOOGLE PAPER ON MACHINE LEARNING & DATA
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 15
If data replaces code in ML
systems, […] then […] testing
of input data is critical to a
well-functioning system.
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
FREQUENT QUALITY ISSUES
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 16
http://blogs.yis.ac.jp/20hepburng/2016/05/24/english-unit-3-reflection-perspective-and-prejudice-english/
FREQUENT QUALITY ISSUES
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 17
Note:
An unbiased and labeling-
independent QA workflow
helps to improve the overall
dataset quality while
keeping track on the
requirements, latest updates
and doubtful situations as
well as insufficiencies!
AI TrainingDataset
DATASET CREATION PROCESS (WITH QA)
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 18
Oil Rig Refinery
Collect Data
Data
Labeling
Quality
Assessment
Inspection
|✓
AI Developer
AI TrainingDataset
DATASET CREATION PROCESS (WITH QA)
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 19
Oil Rig Refinery
Collect Data
Data
Labeling
Quality
Assessment
Inspection
|✓
AI Developer
THANKS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 20
Marius Reuther, Founder DLQA
DEEP LEARNING QUALITY ASSURANCE
www.dlqa.de
