AI and AR - Amazon S3 · • Streamline config. with Docker • Tensorflow works on iOS 9/5/2019...

AI and ARGodfrey Nolan

01.Experience

Experience

• FieldMap

• Create waypoints

• Rolling shutter vs global shutter

• Automated flight around field

• Tile images on iPad

• Stitch images using OpenCV.

9/5/2019 Counting Sheep 5

9/5/2019 Counting Sheep 6

Experience

• Secure Drones Labs

• No data leakage

• Tamper resistant

• Self destruct button

9/5/2019 Counting Sheep 7

02.Counting Sheep??

Counting Sheep

• Requirements

• Semi-practical example

• Using existing experience

• Move to Drone 2.0

• Answer the age old question‘Do Androids dream ofElectric Sheep’

9/5/2019 Counting Sheep 9

03.Plug and Play

Components

• DJI Mobile SDK

• Google Tensorflow

• Google Cloud Platform

• Docker

9/5/2019 Counting Sheep 11

Components

• DJI Mobile SDK

• Google Tensorflow

9/5/2019 Counting Sheep 12

Components

• DJI Mobile SDK

• Google Tensorflow

9/5/2019 Counting Sheep 13

Components

• DJI Mobile SDK

• Google Tensorflow

9/5/2019 Counting Sheep 14

Components

• DJI Mobile SDK

• Google Tensorflow

9/5/2019 Counting Sheep 15

Components

• DJI Mobile SDK

• Google Tensorflow

9/5/2019 Counting Sheep 16

Components

• DJI Mobile SDK

• Google Tensorflow

9/5/2019 Counting Sheep 17

04.Machine Learning Workflow

Workflow

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 19

Workflow

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 20

Workflow

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 21

Workflow

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 22

Workflow

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 23

Workflow

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 24

Workflow

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 25

05.Implementation

Implementation

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 27

Implementation

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 28

Implementation

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 29

Implementation

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 30

Train and Test

• Prepare the Dataset

• Set Up Google Cloud Account

• Set Up Docker

• Configure Google Cloud

• Set Up Object Detection API

• Train Your Model

• Evaluate Your Model

9/5/2019 Counting Sheep 31

Train and Test

Train and Test

• Prepare the Dataset

• Set Up Google Cloud Account

• Set Up Docker

• Configure Google Cloud

• Set Up Object Detection API

• Train Your Model

• Evaluate Your Model

9/5/2019 Counting Sheep 33

Train and Test

• Prepare the Dataset

• Set Up Google Cloud Account

• Set Up Docker

• Configure Google Cloud

• Set Up Object Detection API

• Train Your Model

• Evaluate Your Model

9/5/2019 Counting Sheep 34

Train and Test

• Prepare the Dataset

• Set Up Google Cloud Account

• Set Up Docker

• Configure Google Cloud

• Set Up Object Detection API

• Train Your Model

• Evaluate Your Model

9/5/2019 Counting Sheep 35

Train and Test

• Prepare the Dataset

• Set Up Google Cloud Account

• Set Up Docker

• Configure Google Cloud

• Set Up Object Detection API

• Train Your Model

• Evaluate Your Model

9/5/2019 Counting Sheep 36

Train and Test

• Prepare the Dataset

• Set Up Google Cloud Account

• Set Up Docker

• Configure Google Cloud

• Set Up Object Detection API

• Train Your Model

• Evaluate Your Model

9/5/2019 Counting Sheep 37

Train and Test

• Prepare the Dataset

• Set Up Google Cloud Account

• Set Up Docker

• Configure Google Cloud

• Set Up Object Detection API

• Train Your Model

• Evaluate Your Model

9/5/2019 Counting Sheep 38

Train and Test

• Prepare the Dataset

• Set Up Google Cloud Account

• Set Up Docker

• Configure Google Cloud

• Set Up Object Detection API

• Train Your Model

• Evaluate Your Model

9/5/2019 Counting Sheep 39

Train and Test

• Prepare the Dataset

• Set Up Google Cloud Account

• Set Up Docker

• Configure Google Cloud

• Set Up Object Detection API

• Train Your Model

• Evaluate Your Model

9/5/2019 Counting Sheep 40

Implementation

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 41

Implementation

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 42

Implementation

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 43

Implementation

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 44

Implementation

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 45

Implementation

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 46

Implementation

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 47

Implementation

• Get Data

• Train & Test

• Adjust

• Deploy

• Test & Validate

9/5/2019 Counting Sheep 48

06.Resources Q&A

Lessons Learned

• Crowdsource earlier

• Choose more practical example

• Create your own images

• Image quality

9/5/2019 Counting Sheep 51

Lessons Learned

• Dyed sheep

• Streamline config. with Docker

• Tensorflow works on iOS

9/5/2019 Counting Sheep 52

Future Plans

• Cattle

• Horses

• Deer

• Polar Bear (thermal)

• Parking Lots

• Plants

• Pallets

9/5/2019 Counting Sheep 53

Future Plans

• Animal Tracking

• AR and AI

• Amazon’s Mechanical Turk

• Automate

9/5/2019 Counting Sheep 54

Resources

• http://developer.dji.com

• https://www.tensorflow.org/install/

• https://github.com/tzutalin/labelImg

• https://github.com/tensorflow/models/tree/master/research/object_detection

• https://becominghuman.ai/

• https://towardsdatascience.com/how-to-train-your-own-object-detector-with-tensorflows-object-detector-api-bec72ecfe1d9

• https://docs.bazel.build

• https://t.co/uUTWuPw1ve (whitepaper) and https://github.com/nickdonnellyriis/sheep_demo (code)

9/5/2019 Counting Sheep 55

Q&A

• godfrey@riis.com

• @godfreynolan

• riis.com/blog

• slides.com/godfreynolan

9/5/2019 Counting Sheep 56

07.Extras

Prepare the Dataset

• Gather images

• Label the images with LabelImg found at https://github.com/tzutalin/labelImg

• Generate CSV files from your labeled dataset using script found at https://github.com/datitran/raccoon_dataset/blob/master/xml_to_csv.py

• Convert the CSV files into your TFRecord Dataset using script found at https://github.com/datitran/raccoon_dataset/blob/master/generate_tfrecord.py

• Create your label map file.

9/5/2019 Counting Sheep 58

Prepare the Dataset

9/5/2019 Counting Sheep 59

Set Up Google Cloud Account

• Sign up for a Google Cloud Account

• Set up a project named CountingSheep

• Enable ML Engine for your project https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com

• Click “Storage” on the side bar, and then create a new bucket called CountingSheepData

• Create a sub-directory called “data” in your storage bucket

9/5/2019 Counting Sheep 60

Set Up Your Docker Environment

• Download Docker from docker.io

• Download the Dockerfilehttps://github.com/tensorflow/models/blob/master/research/object_detection/dockerfiles/android/Dockerfile

• docker build .

• docker images

• docker run -it IMAGE_ID

9/5/2019 Counting Sheep 61

Configure your local Google Cloud environment

• Login - gcloud auth login

• Set your google cloud project - gcloud config set project YOUR_PROJECT_NAME

• Set your storage bucket - gsutil mb gs://YOUR_UNIQUE_BUCKET_NAME

• Export Google Cloud project/bucket names - export PROJECT="YOUR_PROJECT_ID"

export YOUR_GCS_BUCKET="YOUR_UNIQUE_BUCKET_NAME"

9/5/2019 Counting Sheep 62

Configure your local Google Cloud environment

• Get the name of your service account:-curl -H "Authorization: Bearer $(gcloud auth print-access-token)" https://ml.googleapis.com/v1/projects/${PROJECT}:getConfig

• Using the tpuServiceAccount value from the response. Export the TPU service account with the following command:-

- export TPU_ACCOUNT=your-service-account

• Grant ML service agent to TPU service account:-gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:$TPU_ACCOUNT --role roles/ml.serviceAgent

9/5/2019 Counting Sheep 63

Set up Object Detection API

• Ensure your local copy of object detection API works with following commands:cd /tensorflow/models/researchpython object_detection/builders/model_builder_test.py

9/5/2019 Counting Sheep 64

Set up Object Detection API

• Move your TFRecord files and label map from your host machine into your docker imagedocker cp train.record CONTAINERNAME:/train.recorddocker cp test.record CONTAINERNAME:/test.recorddocker cp sheep_label_map.pbtxt CONTAINERNAME:/ sheep_label_map.pbtxt

• Copy the data over to your bucket (from your docker container)gsutil -m cp -r /train.record gs://${YOUR_GCS_BUCKET}/data/gsutil -m cp -r /test.record gs://${YOUR_GCS_BUCKET}/data/gsutil cp /sheep_label_map.pbtxt gs://${YOUR_GCS_BUCKET}/data/

9/5/2019 Counting Sheep 65

Set up Object Detection API

• Download the object detection model:curl –O http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_0.75_depth_300x300_coco

14_sync_2018_07_03.tar.gztar xzf ssd_mobilenet_v1_0.75_depth_300x300_coco14_sync_2018_07_03.tar.gz

• Upload it to Google cloudgsutil cp /tmp/ssd_mobilenet_v1_0.75_depth_300x300_coco14_sync_2018_07_03/model.ckpt.*

gs://${YOUR_GCS_BUCKET}/data/

9/5/2019 Counting Sheep 66

Set up Object Detection API

• Update the following values in the config file:/tensorflow/models/research/object_detection/samples/configs/ssd_mobilenet_v1_0.75_depth_quantized_300x300_pets_sync.config

Upload file to Google Cloud bucket with the following command:gsutil cp

/tensorflow/models/research/object_detection/samples/configs/ssd_mobilenet_v1_0.75_depth_quantized_300x300_pets_sync.config gs://${YOUR_GCS_BUCKET}/data/pipeline.config

• Package the object detection API with the following commands:bash object_detection/dataset_tools/create_pycocotools_package.sh /tmp/pycocotoolspython setup.py sdist

9/5/2019 Counting Sheep 67

Train your model

• Start your training job on Google Cloud TPU with the following command:cd /tensorflow/models/researchgcloud ml-engine jobs submit training `whoami`_object_detection_`date +%s` --job-dir=gs://${YOUR_GCS_BUCKET}/train --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-

0.1.tar.gz,/tmp/pycocotools/pycocotools-2.0.tar.gz --module-name object_detection.model_tpu_main--runtime-version 1.9 --scale-tier BASIC_TPU --region us-central1--model_dir=gs://${YOUR_GCS_BUCKET}/train --tpu_zone us-central1 --pipeline_config_path=gs://${YOUR_GCS_BUCKET}/data/pipeline.config

•9/5/2019 Counting Sheep 68

Train your model

• Start your training job on Google Cloud TPU with the following command:cd /tensorflow/models/researchgcloud ml-engine jobs submit training `whoami`_object_detection_`date +%s` --job-dir=gs://${YOUR_GCS_BUCKET}/train --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-

0.1.tar.gz,/tmp/pycocotools/pycocotools-2.0.tar.gz --module-name object_detection.model_tpu_main--runtime-version 1.9 --scale-tier BASIC_TPU --region us-central1--model_dir=gs://${YOUR_GCS_BUCKET}/train --tpu_zone us-central1 --pipeline_config_path=gs://${YOUR_GCS_BUCKET}/data/pipeline.config

•9/5/2019 Counting Sheep 69