Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
ISAACAn Accelerated Robotics Framework
Hammad Mazhar
2
Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine
PR2 - HOCKEY
3
PROJECT ISAAC: HOCKEY
Teach a robot how to play hockey using ground truth
Robot: PR2 (Willow Garage)
Kinematic model
7 joints, 1 arm
Observations:
Puck and Joint positions, velocities
Goal location
3
4
PROJECT ISAAC
5
DISCO HOCKEY BOT
5
6
ISAAC LAB: DOMINOES
6
7
SYNTHETIC DATA GENERATION• Generate 50K images
• Multiple distractors, lights, etc
• Labeled:
• Location, pose, pip count
7
8
GAMEPLAY LOGIC TRAINING (SLOW MOTION)
8
9
ISAAC PLAYS DOMINOES
10
ROBOTICS AT NVIDIATransfer learning in Isaac Lab
11
ISAAC PLAYS GOLF
12
VIRTUAL DEMONSTRATION FOR ROBOTICS
13ISAAC SDK
14
ISAACSDK FOR ROBOTICS
Simulation EnginePhoto-realistic Graphics ∙ Physics ∙
∙ Procedural Generation ∙ Massive parallelism
∙ Unreal Engine 4
Isaac FrameworkCodelets ∙ Behaviors ∙ 3D Poses ∙ Distributed
∙ Messaging ∙ Synchronization ∙ Record & Replay
∙ Configuration ∙ Visualization
World modelWarehouse ∙ Office
∙ Store ∙ Home
Robot modelCarter ∙ URDF loader
Virtual Sensors Virtual Actuators Sensor Processing Actuator Control HW Sensor HW Actuator
MLTensorRT ∙ CUDA
∙ Tensorflow ∙ ...
GemsOptimizers ∙ Algebra
∙ EKFs ∙ Depth ∙ ...N
avig
atio
n
Be
ha
vio
rs
Inte
ractio
ns
with
hu
ma
ns
Manip
ula
tion
Pe
rce
ptio
n
DriversLidar ∙ Camera ∙ IMU ∙
Robot Base ∙ ...
Unified Message APIUse the same messages for simulation,
actual hardware and across all apps
Simulate DeployDevelop
JetsonFully integrated with
X2 and Xavier
15
ISAAC SDK: A ROBOTICS TOOLBOX
Jetson AGX XAVIER
Advanced GPU, 64-bit CPU, Video CODEC, VIC, ISP
CUDA, Linux4Tegra
Core libraries Navigation Reference DNN
VisionWorks, OpenCV, TensorRT, cuDNN
ToolsDrivers
16
NODE BASED PROGRAMMING INTERFACE
17
Global planner
NAVIGATION GEMS
Global planner Obstacle avoidance Localization State machine
18
STEREO DNN
19
DEVELOPER TOOLS
20
ISAAC SIMULATION
21
SENSOR SIMULATIONCameras and Lidar
22
MANIPULATIONURDF Support
23
ROSInteroperability with existing robotic stacks
24
25
DESIGNING MEANINGFUL TRAINING ENVIRONMENTS
25
26
DESIGNING MEANINGFUL TRAINING ENVIRONMENTS
26
27
PHYSX 4.0
28
CARTER
29
CARTER REFERENCE DESIGN
Velodyne VLP16 Zed stereo IMUSegway
Carter
Jetson AGX Xavier
30
Iterate at high speed on a desktop and in simulation
Deploy and test in reality
Training on DGX
CARTER REFERENCE DESIGN
Xavier and TX2
Carter app
31
CARTER WITH ISAAC SDK
32
SALAD DELIVERY
33
NEXT STEPS
34
July 2018 2019 2020
CARTER IS JUST THE BEGINNING