JUNE 21-25, 2021

REALSIM

DEAN DETERPrinciple Investigator Vehicle Systems Integration Research GroupOak Ridge National Laboratory

PROJECT ID: EEMS101

PROJECT OVERVIEW

2

Timeline Barriers

Budget Partners

• Project Start Date: Oct. 2020

• Project End Date: Sept. 2023

• Percent Complete: 10%

• Total Funding: $3.58M

• Funding per Year:

• FY 21: 1.12M

• FY 22: 1.23M (Planned)

• FY 23: 1.23M (Planned)

• Lead

• Oak Ridge National Laboratory

• Partner

• Argonne National Laboratory

• Collaborations

• American Center for Mobility (ACM)

• IPG Automotive

• Modeling and simulation environments are not

all inclusive for all scenarios.

• Lack of standard co-simulation tools or hooks

across vehicle and traffic environments.

• Computational requirements of complex

environment simulation.

OBJECTIVES AND RELEVANCE▪ Relevance:

1. A system of systems approach is required to properly manage vehicles equipped with advanced

driver-assistance systems (ADAS), connected and autonomous vehicle (CAVs) technologies, their

interactions with the surrounding environments, traffic networks, traffic control, and the various

scenarios that greatly effect each one of these.

2. Expensive in-field experimentation generally captures ‘focused’ datasets- not providing sufficient

detail to produce a digital twin or explain unique events during future experimental analysis.

Current on-road connectivity implementations are built upon large infrastructure investments at

permanent locations. (Corridors / test tracks / etc.)

▪ Objective: 1. Expand the capabilities of the VPPG Core-tools project feedback loop to include capabilities of

digital twins as well as greater real-world correlation by utilizing on-road/track data in a closed-loop

process for validation of the performance of controls and simulated scenarios/environments.

2. Incorporating joint vehicle and infrastructure experimental platforms can provide a “full picture” of

costly experiments for further analysis OR replication in a digital environment where more

variables may be controlled. Smaller scale, portable systems built on ‘nodes’ offer flexibility and

control of research activities, enabling experiments on controlled roadways, or public roads.

3

MILESTONES

4

Date Milestone Status

June 2021

Demonstration of sensor simulation and integration into

ORNL/ANL ROS/Autoware based perception stack including

camera, radar, and lidar systems.

(Task 1.1 ORNL)

On Track

June 2021

Demonstration of microtraffic simulation’s traffic control

functionality and integration into Real-Sim including

SPaT control and higher-level traffic control.

(Task 1.2 ORNL)

DelayedDue to COVID

shutdowns and

limited NRTL

inspection

resources.

Sept. 2021

Demonstration of ANL prototype platform with vehicle

and infrastructure components.

(Task 1.3 ANL)

On Track

APPROACH

5

Direct support through data for multiple

SMART 2.0 efforts

Develop a mobile connectivity-enabled

platform to coordinate and capture on-

road experimentation

Vehicle Component

Infrastructure Component

Capturing ‘full picture’

vehicle operation AND

environment during

experimentation

Providing control and

data capture of traffic

flows and infrastructure

components

APPROACH

▪ Subtask 1.1: Sensors - XIL and

virtual environment 2.0.

▪ Subtask 1.2: VPPG microtraffic

2.0, enhancements to include

SPaT, V2X, and traffic control

XIL applications.

▪ Subtask 1.3: Development of

portable platform for V2X

testing and data collection to

enable XIL validation.

6

Task 1 Task 2 Task 3

▪ Task 2 – Real-Sim platform and

digital twins

▪ Digital Twins to be created for

support of this project and

SMART 2.0

– Multiple areas of

Chattanooga, TN

– ACM (improvement of digital

twin)

– Lake County, IL

▪ Task 3 – Validation of the Real-

Sim Platform using Current

On-Road/Track EEMS Projects

▪ Projects that will support this

effort.

– Regional Mobility Project

– ACM Model Validation

Project

– EEMS096

– EEMS095

– EEMS061

ACCOMPLISHMENTS

7

TECHNICAL ACCOMPLISHMENTS

▪ IPG Carmaker has a ROS interface that

bridges communication between the

simulated sensors and a specific ROS topic

▪ This interface allows the user to build up a

message and stream information to a

specific topic over the ROS network

▪ The initial interface had a lidar and camera

streaming data over ROS. The interface

was extended to include a GPS and IMU

as well.

Task 1.1: Adding sensors to ROS data streaming from IPG Carmaker

8

Msg: gps_common/GPSFix

Topic: /fix

Msg: sensor_msgs/Imu

Topic: /imu

TECHNICAL PROGRESS

▪ The digital twin of the ORNL main

campus in IPG Carmaker imported to

this Carmaker project. Also, a visual

model of the ORNL RAV4 mule

vehicle was imported to Carmaker

▪ This setup deploys the ORNL mule

vehicle onto the digital twin of the

main campus running a defined

scenario. This scenario can match

the real mule vehicle data for better

correlation.

Task 1.1: Scenario and vehicle development in IPG Carmaker

9

TECHNICAL PROGRESS

▪ The sensor setup on the vehicle was

modified to match a subset of the

mule vehicle sensors (1 lidar, 1

camera w/ object list, gps and imu).

▪ The sensor streams can then be

visualized in tools that listen to ROS

messages (Rviz, command line tools,

etc.)

Task 1.1: Modification of vehicle sensor streams from IPG Carmaker

10

TECHNICAL PROGRESSTask 1.1: Video of all systems executing together in IPG Carmaker

11

▪ Determined key signal list from collaborative efforts

– Experiment focused data collection

• Raw sensor data collection for digital twin

development from focused in-field experiments

• Condensed vehicle and object data for vehicle

and traffic impact analysis, reducing dataset size

▪ Development of platforms for vehicle component

– Vehicle selection based on collaborative studies

– Decoded vehicle data from CAN / FlexRay

– Hardware modifications for sensor array

– Development of ROS-based platform for time

aligned data collection of vehicle and sensor data

▪ Infrastructure component development - Q3/Q4

AMTL Mobile DAQ

TECHNICAL ACCOMPLISHMENTS

2020 Tesla Model 3AutoPilot / FSD

2019 Cadillac CT6SuperCruise

Transferable Mobile DAQ – independent sensor /

environment data

• Video (multi camera)

• LiDAR (Velodyne VLP-16 x2 / VLP32C x2 )

• Communication- (DSRC)

• Plugins for time alignment with recorded vehicle data

RESPONSE TO PREVIOUS YEARREVIEWERS’ COMMENTS

▪ This is the first year that the project has been reviewed.

13

PARTNERSHIPS ANDCOLLABORATIONS

14

Primary project partner and contributor for Task 1.3 focused

on the prototype DAQ unit for vehicles and infrastructure.

This system is key for the completion of Task 2 and 3.

EEMS096, EEMS095, EEMS067, EEMS061

Digital twin of ACM facility was provided to ORNL to allow for

a digital twin to be generated and utilized. Additionally, data

collected from ACM-led EEMS082 is provided to ORNL

IPG is collaborating with ORNL via software support for

sensor modeling and emulation, as well as being a key

partner is working with ROS and Autoware for XIL

applications.

REMAINING CHALLENGES AND BARRIERS

▪ Though research facilities are open for experimentation, controls are in place to ensure

safety during COVID-19 pandemic which impact worker productivity and operations.

Reduced staff available onsite, in addition to the needs for social distancing, can make

hands-on research and troubleshooting efforts challenging and more time consuming.

Though no milestones or objectives are expected to be missed at this time, the possibility

exists due to resource constraints. Creative methods for on-site and remote work will

continue throughout limited, or reduced, site operations and the risk of impacts to project

timing remain.

▪ Determining the computational requirements for these various applications is a major

challenge as well as a barrier to some applications. To run the most demanding sensor

simulations (i.e. lidar and radar) on HIL applications requires an extremely powerful PC to

apply ray tracing techniques utilizing GPU calculation and a powerful real-time node.

15

PROPOSED FUTURE RESEARCH

▪ Remainder of 2021– Continue work on sensor simulation and data stream

emulation to vehicle sensors.

– Demonstration of ANL prototype platform with vehicle

and infrastructure components functional.

▪ Future FY 2022-2023– Initial trial, refinement, and further deployment of field

components for AMTL Mobile DAQ

– Construct the digital twin of the Lake County intersection

in Illinois from scratch

– Design scenarios based on tests from 1591, 1562, and

SMART Mobility Validation at ACM on the Real-Sim

platform to generate comparison data for correlation with

road/track data.

Any proposed future work is subject to change based on funding levels.

16

SUMMARY

17

ApproachRelevance

Future Work Technical Accomplishments

• The importance of the system of systems approach for

understanding and managing vehicles equipped with

ADAS, CAVs technologies, and their interactions with the

surrounding traffic environments.

• Field experimentation captures ‘focused’ datasets lacking

detail for detailed experimental analysis. Current

connectivity implementations rely on large, fixed

infrastructure investments.

• Task 1:

• Sensors - XIL and virtual environment 2.0.

• VPPG microtraffic 2.0

• AMTL Mobile DAQ

• Task 2: Real-Sim platform and digital twins

• Task 3: Validation using Current On-Road/Track EEMS

Projects

• FY 2021:

• Sensor simulation and data stream emulation

• Demonstration of AMTL Mobile DAQ platform

• FY 2022/FY2023:

• AMTL Mobile DAQ initial trial, refinement, and

further deployment

• Lake County digital twin construction

• EEMS scenario execution and correlation

• Sensor data streaming using ROS

• Vehicle and scenario development in IPG Carmaker

• Modification of vehicle sensor streams

• Determined key signal list from collaborative efforts

• Development of platforms for vehicle component

MOBILITY FOR OPPORTUNITY

FOR MORE INFORMATION

Dean DeterR&D Staff

Vehicle Systems Integration Research

National Transportation Research Center

Oak Ridge National Laboratory

deterdd@ornl.gov

18

TECHNICAL BACK-UP SLIDES

20