55
http://www.skywatch24.com 雲端影與物聯網平台之 軟體程挑戰 陳維超 Wei-Chao Chen | Skywatch Inc. | National Taiwan University

雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

  • View
    6.682

  • Download
    8

Embed Size (px)

Citation preview

Page 1: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

http://www.skywatch24.com

雲端影⾳音與物聯網平台之軟體⼯工程挑戰

陳維超Wei-Chao Chen | Skywatch Inc. | National Taiwan University

Page 2: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

WARNINGMay contain shameless self-promotion

YOU HAVE BEEN WARNED

Page 3: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

1998 2002 2006 2010 2014

ABOUT MYSELF

Page 4: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

Internet Camera x Cloud Powerwww.skywatch24.com

Page 5: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

extend����������� ������������������  your����������� ������������������  perception����������� ������������������  ����������� ������������������  giving����������� ������������������  meaning����������� ������������������  to����������� ������������������  the����������� ������������������  places

raw data

there����������� ������������������  are����������� ������������������  3����������� ������������������  people����������� ������������������  in����������� ������������������  your����������� ������������������  yard!����������� ������������������  ����������� ������������������  

meaningfulmessage

Page 6: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

Past

RouterNAS

Active

SureConnectcan’t reach

can break

simple����������� ������������������  and����������� ������������������  reliable����������� ������������������  connectivity

Page 7: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

in����������� ������������������  over����������� ������������������  10����������� ������������������  countries����������� ������������������  complete����������� ������������������  hardware����������� ������������������  &����������� ������������������  software����������� ������������������  solution

Page 8: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

cloud����������� ������������������  video����������� ������������������  analytics����������� ������������������  

make����������� ������������������  regular����������� ������������������  network����������� ������������������  cameras����������� ������������������  smart����������� ������������������  !

?…

Page 9: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

318 BROADCASTING

Page 10: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超
Page 11: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

THE RESEARCH

Page 12: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

VIDEO ANALYTICSA methodology to reason and understand video

I saw three person

last hour

Page 13: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

People Counthttps://www.youtube.com/watch?v=TPjmsha6gWY

Loiteringhttps://www.youtube.com/watch?v=f5DxUe5I8as

Trip wirehttps://www.youtube.com/watch?v=MH2HZi6-4jA

Speed testhttps://www.youtube.com/watch?v=KV6CXdzmhNs

Page 14: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

SMART FAST FORWARD

Video activity level

Single button interface

Page 15: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

SMART FAST FORWARD

•20x average speedup

•Compressed domain processing

•30x faster processing compared to competition

•3 seconds for every 10 minutes

Page 16: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

SMART FAST FORWARD

Page 17: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

SMART DAILY•Intelligent video summary

•Daily email on your past day’s activity

- Compressed domain analysis

- Very fast - totally I/O bound

- Detect events based on activity / sensible motion

Page 18: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

Click to see video Yes, it is secure

(How?)

Page 19: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

SMART CAPTION•Image to text / deep learning

Page 20: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

SMART CAPTION•Accuracy not critical for application (?!)

•Deep learning classification

•Berkley Caffe

• ImageNet / Microsoft COCO

• Stanford NeuralTalk

•GPU / CUDA powered

•<1 second per image

Page 21: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

NEURALTALK

http://cs.stanford.edu/people/karpathy/deepimagesent/generationdemo/

Page 22: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

NEURALTALK

http://cs.stanford.edu/people/karpathy/deepimagesent/generationdemo/

Page 23: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

FEATURE MATRIXTr

aini

ng C

ost

Runtime Cost

FaceDetect

LoiteringTrip wire

Area AlertLeft object

Motion

LicensePlate

PeopleDetection

Face

Reco

gnitio

n

Object

Classificat

ion

Age

Gender

PeopleCount

HeatMap

SmartCaption

SmartFast

Forward

SmartDaily

Skywatch

RegularIVA

Page 24: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

EDGE COMPUTATION

Compute The Internet

Video Meta-data

Je suis un Edge

Page 25: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

CLOUD COMPUTATION(SOMEWHAT POINTLESS)

The Internet

Video

Meta-dataCompute

Page 26: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

CLOUD COMPUTATION(SENSIBLE)

Video

Meta-data

The Internet

Compute

Data model Video storage

Page 27: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

HYBRID COMPUTATION(OUR VISION)

Early Vision

Partial Meta-data

The Internet

Late Vision

Data model Video storage

Various meta-data

results

Page 28: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

CLOUD VS LOCALCloud Local

How? Cheap camera Cloud computer

Special camera Local computer

Best for Search & classify Big data Real-time alerts

AdvantagesEasy to deploy & manage

Cheap camera Rental model

Lower bandwidth Real-time applications

Disadvantages Higher bandwidth Maybe not for alerts

Difficult to deploy & manage Expensive camera Higher upfront cost

Page 29: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

THE ENGINEERING

Ivan Sutherland, the Sketchpad

Page 30: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

WE HAVE BUILT STUFFS

•Chief shader architect for Playstation 3 (RSX)

•75% of the GPU chip

•Expensive bugs (> $100K USD each)

•Targeted test, random test, monkey test

•From architecture to driver to hardware

•Shipped over 100M systems

Page 31: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

WE HAVE INVENTED STUFFS

•OpenCV

•MPEG4 Source / OpenLF

•Fancy publications

•US Patents

Page 32: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

HOW HARD CAN IT BE?Get the plumbing done, then add smartness

Page 33: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

Prototype

Scalable & Stable

Initial Production

Prototype

Page 34: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

SOFTWARE ENGINEERING

Aspects What do we do?

Development Scrum / continuous integration

Testing Unit / regression testing

Review Design review / Design patterns Code review

Operation Push-button code deployment Server monitors / camera diagnostic tools

What about the cameras?

Page 35: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

TESTABILITY

Test Subject

Data

Support Units

Interface

Page 36: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

Sandbox (Truman’s world)

TESTABILITY

Test Subject

Test Data

Fake / Mock

Interface

Page 37: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

Sandbox (Truman’s world)

TESTABILITY

Test Subject

Test Data

Fake Camera

Interface

Really hard to make a good fake device

Page 38: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

COMPLEXITY

Firmware version Camera models

Server config Software delivery

Browser type App version

Page 39: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

CONTROL THE COMPLEXITY

OTA upgrade Fewer models

VM / container Push-button release

Versioned API Limit OS revision

Frameworksare yourfriends(?!)

Page 40: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

High-bandwidthvideo traffic

Low-bandwidthAPI traffic

System Architecture

Overview (Dual Region)

Storage SAN

ComputeServers

StreamingServers

Region AP2P

Video

Storage SAN

ComputeServers

StreamingServers

Region B

UserDB

ApplicationServersApplicationServers

Page 41: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

Embedded Linux

Camera Monitors

VPN Interface

P2P Interface

RTSP Streaming

RTMP Streaming

LAN Interface

Camera API

VPN Interface

RESTfulCloud API

User DB

StreamBroadcaster Video Files

Camera Software Stack

Skywatch Service API

Internet

Skywatch Firmware / Software

Camera Firmware

Page 42: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

TOPEARLY SCREW-UPS

Page 43: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

#1: BOB SAW ALICE(AND MAYBE CHARLES)

Alice

BobBob

Alice

Charles

Very early mistake — this won’t happen again

Page 44: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

•How did this happen?

•Not thread-safe / process-safe

•Unique IDs are not unique (after a certain scale)

•Loose DB & network synchronisation

•Old tunnels hijacked by new cameras

•Camera bug — HTTP authentication off in a firmware version

#1: BOB SAW ALICE(AND MAYBE CHARLES)

Page 45: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

Alice

•Why it won’t happen again?

#1: BOB SAW ALICE(AND MAYBE CHARLES)

Alice

Bob

HTTP Password

authenticatedCharles

Individually signed & CRC’ed files

Scanner to make sure camera

authentication is on

Channel scanner to clean up dangling

connections

RESTful API check for signature

Page 46: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

#2: FIRMWARE UPGRADE EQUALS ANGRY CUSTOMERS

•Auto upgrade?

•Bad idea — people unplug and brick the camera

•User-triggered OTA upgrade

•Camera restore to default & WiFi settings gone

•Continuous upgrade testing is needed

•Monitor customer upgrade events / failures

Page 47: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

#3: JAVA - WRITE ONCE,DEBUG EVERYWHERE

•Browser H.264 decoder / multi-channel player

•Originally JAVA-based. Why?

•Because a big company is using it

•Because Flash was killed by Jobs

•Because we can save money (P2P)

•Tons of support issue (!)

Browser Player Flash / HTML5

player

Page 48: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

#3: JAVA - WRITE ONCE,DEBUG EVERYWHERE

•Browser H.264 decoder / multi-channel player

•Originally JAVA-based. Why?

•Because a big company is using it

•Because Flash was killed by Jobs

•Because we can save money (P2P)

Browser Player Flash / HTML5

player

Believe in yourself, not the competitor

Choose a popular weapon

Worry about money saving when it becomes a real issue

Page 49: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

THE OUTLOOK

Preikestolen (cc) Anna Bialkowska https://www.flickr.com/photos/annabialkowska/117380607

Page 50: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

Converse

Control

Connect

Data and services need to talk

to each other

Page 51: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

Soonsensors����������� ������������������  that����������� ������������������  talk����������� ������������������  to����������� ������������������  ����������� ������������������  each����������� ������������������  other����������� ������������������  intelligently

Page 52: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

record����������� ������������������  kid’s����������� ������������������  study����������� ������������������  time����������� ������������������  and����������� ������������������  compare����������� ������������������  with����������� ������������������  her����������� ������������������  score

08:08AM 07:38PM

4hours 2hours

a����������� ������������������  stranger����������� ������������������  sneaking����������� ������������������  around����������� ������������������  in����������� ������������������  your����������� ������������������  neighborhood

7/2 7/7 7/9

?

Page 53: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

retailrescue.com

People Counting v.s.

Stock Market

Page 54: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

Power transmission lines (cc) Oran Viriyincy https://www.flickr.com/photos/viriyincy/7004881064

Home Automation v.s.

Grid Management

Page 55: 雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

http://www.skywatch24.com

THANK YOU陳維超

Wei-Chao Chen [email protected]/weichao.chen.tw

We are hiring!!