Charles Fay, Sr. Program Officer Big Data Meets Computer Vision

Dec. 7, 2012

Accelerating solutions for highway safety and performance

SHRP 2 Strategic Highway Research Program

Like challenges? Then you should be excited by SHRP 2 NDS

~ 4 petabytes of data that need to be post-processed ~ 1 million hrs of video ~ 3000 subjects, 5 million trips, >18 million miles driven,

4 billion GPS points Real world - automotive conditions (daylight

variance; nighttime IR); low quality cameras & images Data compressed(H264) and saved at 15 Hz  PII (personal identifiable information) & protection of

privacy Patience with getting access to data- working out details

 

Advisors to the Nation on Science, Engineering, and Medicine

To: "investigate, examine, experiment, and report upon any subject of science or art - whenever called upon to do so by any department of the government”

Transportation Research Board (TRB) is one of six major divisions

Est. 1863

National Academy of Sciences

Content

•What’s the Problem(s)? •Preview video data •Naturalistic Driving Study (NDS)•Roadway Information Database(RID)•FHWA Exploratory Advanced Research Program •Goal today:

•promote interest in mining these datao making these data more usable

Ultimately saving lives/ reducing severity of injury

Public Health & Highway Safety: Crashes leading cause of death

for ~ 4-34 year old (US)*

~ 40,000 total deaths in US/year*

~ 2.5-3.0 million injuries /yr in US Estimated costs: $230 billion/ yr in US

Driver behavior has been identified as the major factor in 90% -95% of roadway crashes (know very little about behavior )

Major issue around the world; Naturalistic driving studies in EU, China, Australia; others in development- way of the future

wot.motortrend.com

Computer Vision: Before analysts can use the full NDS dataset –

more usable form – that is where you come in Lots of data from ~ 3000 participants

▪ ~ 4 petabytes; 1 million hrs video + other sensor data; 5 million trips; > 18 million miles

Saved video poorer quality relative to what you are used to analyzing.

PII (personal identifiable information) & data access (working out details-patience please)

▪ recording continuously: GPS; face video

DRIVER RELATED

Driver behavior

Distraction

Head pose

Eye gaze

Fatigue/drowsiness

Mobile device use

Hand position

Foot/pedal

CONTEXT RELATED

Traffic signal state

Roadside information

Weather, pavement

conditions

Bike/ Pedestrian

Other vehicles (brake

lights) & traffic

“Your challenge should you choose to accept…

…working on post processing these data in an efficient manner to gain meaningful information”

kmnnz.wordpress.comhttp://kellypuffs.wordpress.com

Benefits of the Study (safety related) These data are not available – one of a kind database(s):

decades of use

Almost Everyone (OEMs, DOTs, researchers) eager to get hands on these data

• Intelligent / automated/connected vehicles & transportation

• Improved understanding of baseline driving behaviors: Trip characteristics Driver performance profiles Adherence to laws and basic safety practices

• Improved understanding of unsafe behaviors and traffic events:

Assess circumstances and motivations for speeding, red light running, etc. Deconstruct crashes and near-misses and examine causality How do driver, vehicle, roadway, and environmental factors influence behavior and

impact crash risk?

• Improved ability to develop safety countermeasures for:

Education and training Roadway design and traffic engineering Vehicle design Regulation and enforcement Ability to direct countermeasures at driver subgroups

Camera Image SamplesCamera Image Samples

Center stack – Pedal Interactions; hands

Forward View - color

Right-Rear View

Driver Face – Rotated for max pixel efficiency

Periodic still cabin image, permanently blurred for passenger anonymity (child safety seat use?)

What can be done post-processed?What can be done post-processed?

Video saved @ 15Hz; H 264 compression Video saved @ 15Hz; H 264 compression

11

480x360Scaled full to 480x360

240x

360

Scal

ed fu

ll to

360

x240

or c

ropp

ed a

t 36

0x24

0 an

d Ro

tate

d 90

deg

rees

360x120Scale Vertical by ¼ horizontal by 1/2

360x120Crop 25% off top and Bottom then Scale by 1/2

Camera view

Horizontal FOV sensor lens Lines effective

pixels stored size

Image alterations Measured

Forward view 82 1/3 DPS

CMOS F= 3.6/F1.4 540 720 (H) X 540 (V) 480X360 Full scaled 83.5

Face 75 1/4" BW CCIQ II camera

3.3mm/F2.0 400 648X488 (EIA/NTSC) 240X360

full scaled rotate 90 degrees 78

Rear 92 1/4" BW CCIQ II camera

2.1mm/F2.5 400 648X488 (EIA/NTSC) 360X120

Crop 25% off top and Bottom then Scale by 1/2

100

Instrument Cluster 92

1/4" BW CCIQ II camera

2.1mm/F2.5 400 648X488 (EIA/NTSC) 360X120

Scale Vertical by 1/4 horizontal by

1/2 95

Cabin 92 1/4" BW CCIQ II camera

2.1mm/F2.5 400 648X488 (EIA/NTSC) Blurred Jpeg 92

NDS RID

NDS Data

RID (GIS)

(DAS GPS is Link)

Existing Datacharacterize the

environment in which the participant/ DAS operates:

roadway, crash, safety campaigns,

laws, traffic, weather, work

zones…linked to roadway

segment

New Roadway Data Collected and QA

~ 1950 DAS~3000 participants~ 5 million trips

Passenger Car, Van, SUV, Pickup

Six NDS Data Collection Sites across the U.S.

One Coordinator

NDS Data

WAData Collection

INData Collection

NCData Collection

NYData Collection

PAData Collection

FLData Collection

NDS DataNDS Data Driving data: from instrumentation on vehicle

Driver data: from questionnaires, tests

Vehicle data: vehicle inspection; CANbus-vehicle network

Crash data: detailed investigation of selected crashes

Will include both restricted and non-restricted data

requiring various levels of access

Restricted data: that which may be used to identify a

participant, such as face video or GPS. Requires high level

of physical and electronic security, data access

agreements, ethics review, oversight. Working on

specifics for data access (remote enclave(s) being

considered)

Non-restricted data can be disseminated more widely via

web access, summarized data sets, numerical variables

18

Multiple VideosMachine Vision

Eyes Forward Monitor Lane Tracker

Accelerometer Data (3 axis) Rate Sensors (3 axis)GPS

Latitude, Longitude, Elevation, Time, Velocity

Forward Radar X and Y positions X and Y Velocities

Cell Phone ACN, health checks,

location notification Health checks, remote

upgrades

Illuminance sensorInfrared illuminationPassive alcohol sensorIncident push button

Audio (only on incident push button)

Turn signalsVehicle network data

Accelerator Brake pedal activation ABS Gear position Steering wheel angle Speed Horn Seat Belt Information Airbag deployment Many more variables…

DAS Overview

20

3500-3900 total vehicle years

Thru Lane: 1(21’)

Thru Lane: 1 (12’)Accel. Lane: 1

Thru Lane: 2 (11’)Deccel. Lane: 1

Thru Lane: 1 (12’)Left Turn Lane: 1

ThruLane: 1 (14’)

Deccel. Lane: 1

FlushPaint.

Flush Paint.

Flush (Painted)

Flush (Painted)

2’ Mix/Combo 0’ Mix/Combo 3’ Mix/Combo 2’Mix/Combo

N/A N/A N/A

Grade, Cross Slope

Unpaved Shoulder: N/A

Rumble Strips: N/ALighting: N/A

FlushPaint.

Flush Paint.

Flush (Painted)

Flush (Painted)

N/A N/A N/A

Thru Lane: 1 (12’) Thru Lane: 1 (11’)Right Turn: 1

Thru Lane: 1 (12’)

3’ Mix/Combo 3’ Mix/Combo 4’ Mix/ComboN/A

Paved Shoulder

Median

Lanes

Paved Shoulder

Median

Lanes

•Horizontal Curvature: Radius , Length ,PC , PT ,Direction

•Grade

•Cross Slope/ Super Elevation

•Lane in terms of the number, width, and type ( turn, passing,

acceleration, car pool, etc…)

•Shoulder type/curb; paved width if exists

•Intersection location , number of approaches, and control (uncontrolled,

all-way stop, two-way stop, yield, signalized, roundabout). Ramp termini

are considered intersections

•Posted speed limit sign and location (R2-4 Series)

•Median presence(Y/N), type (depressed, raised, flush, barrier)

•Rumble Strip presence(Y/N) location (centerline, edgeline, shoulder)

•Lighting presence( Y/N)

•FHWA determining if additional data types will be processed (e.g., All

MUTCD signs; barriers - TBD)

Route Name Direction Chainage State Collection Date

Front ROW Images

# Item Priority1 Crash Data

1

2 Traffic Information - AADT

8 Aerial Imagery 9 Speed Limit Data

10 Speed Limit Laws

11 Cell phone and text messaging laws

12 Automated enforcement laws

13 Alcohol-Impaired and Drugged Drivers laws

14 Graduated driver licensing (GDL) laws

15 State motor cycle helmet use laws

16 Seat Belt Use laws

5 Local Climatological Data (LCD) NOAA

17 Cooperative Weather Observer/Other Sources

4 Winter Road Conditions (DOT)

2

3 Work Zone24 511 Information

18 Traffic Data - Continuous Counts (ATR)

19 Traffic Data -Short Duration Counts

21 Changes to existing infrastructure condition

22 Roadway Capacity Improvements

6 Nonrecurring Congestion 320 Automated Enforcement

7 Travel Time Data

23 Innovative Treatments 4

25 Recurring Congestion

25

Working on providing data from 24 individuals

~ 45 min per driverVariety of facial featuresGlasses/sunglassesDaytime/ nighttime conditions IRB; consent form allow data to be

shared for research purposes May need your IRB approval- most likey

expedited review

Exploratory Advanced Research Program

Video analytics workshop: 10/10-11/2012 summary report by January 2013 http://www.fhwa.dot.gov/advancedresearch/

David.Kuehn@dot.gov Lincoln.Cobb@dot.gov

Charles Faycfay @nas.edu202-334-1817