Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Electrical & Computer Engineering Department
The University of Tennessee, Knoxville334 Ferris Hall, Knoxville, TN, 37996-2100, USA 1
Imaging,Robotics, &IntelligentSystemsLaboratory
Imaging,Robotics, &IntelligentSystemsLaboratory
DOE-URPR Project
3D Imaging and Data Fusion in Robotics
Mongi Abidi
Professor
Director of IRIS
Associate Department Head
Electrical and Computer Engineering
University of Tennessee, USA
http://imaging.utk.edu
Electrical & Computer Engineering Department
The University of Tennessee, Knoxville334 Ferris Hall, Knoxville, TN, 37996-2100, USA 2
UTENN - IRIS Laboratory
Prof. M. A. Abidi(Group Leader)
Prof. A. KoschanProf. J. PaikDr. A. GribokDr. B. R. Abidi
Dr. D. Page
PARTICIPANTS
$21,000,000+ since 87
Sponsors: DOE, DOD, NSF, TACOM, NASA, TVA, L-Martin, TI, Radian, Eastman Chemical
Company, National Automotive Center
FUNDING
• 30,000 sq. feet of Lab space• Workstations: Sun, SGI, PC
•Sensors: Vision, Range, etc...
FACILITIES
Faculty: 70+ man-years of Exper.in Image Processing & Robotics
Staff of 30 (17 Ph.D./MS/UG Students)
300 Journal/Conference Papers
8 Textbooks/Reference Volumes
ACCOMPLISHMENTS
UT- IRISLab
UT- IRISLab
Electrical & Computer Engineering Department
The University of Tennessee, Knoxville334 Ferris Hall, Knoxville, TN, 37996-2100, USA 3
Robot System Communication
and control
High-level Reasoning
Trajectory Planning
Motion Control
End-effector Control
Sensor Activation
Data Acquisition
Vision
Range
Sound
Proximity
Force/Torque
Touch
Interface Functions for Autonomous Robotic Inspection and Manipulation
Electrical & Computer Engineering Department
The University of Tennessee, Knoxville334 Ferris Hall, Knoxville, TN, 37996-2100, USA 4
IRIS Academic Program
Senior Level: 405 Digital Signal Processing and Filter Design451 Microprocessors in Computer Engineering471 Introduction to Pattern Recognition472 Introduction to Digital Image Processing494 Special Topic in Image Processing and Robotics
Master’s Level: 503 Modern Transform Methods504 Random Process Theory for Engineers505 Digital Signal Processing I506 Digital Signal Processing II571 Pattern Recognition572 Digital Image Processing573 Vision Sensing for Robotics & Automation I574 Vision Sensing for Robotics & Automation II
Ph.D. Level: 671 Image Processing and Robotics I672 Image Processing and Robotics II673 Image Processing and Robotics III
Senior Level: 405 Digital Signal Processing and Filter Design451 Microprocessors in Computer Engineering471 Introduction to Pattern Recognition472 Introduction to Digital Image Processing494 Special Topic in Image Processing and Robotics
Master’s Level: 503 Modern Transform Methods504 Random Process Theory for Engineers505 Digital Signal Processing I506 Digital Signal Processing II571 Pattern Recognition572 Digital Image Processing573 Vision Sensing for Robotics & Automation I574 Vision Sensing for Robotics & Automation II
Ph.D. Level: 671 Image Processing and Robotics I672 Image Processing and Robotics II673 Image Processing and Robotics III
Electrical & Computer Engineering Department
The University of Tennessee, Knoxville334 Ferris Hall, Knoxville, TN, 37996-2100, USA 5
Sensory Data Acquisition
IRIS Range Sensing Platform: Perceptron-based 3D Laser finder
High-resolution Color Camera
(Kodak 3000x2000 pls)
Point/Line Scanning Laser Range System
assembled at IRIS lab.
Electrical & Computer Engineering Department
The University of Tennessee, Knoxville334 Ferris Hall, Knoxville, TN, 37996-2100, USA 6
From IMAGES to 3D MODELS
3D models containing information such as visual, geometrical, thermal, radiometric, etc.
INPUT OUTPUTRange imagesDigital elevation maps 3D models
Electrical & Computer Engineering Department
The University of Tennessee, Knoxville334 Ferris Hall, Knoxville, TN, 37996-2100, USA 7
Functional Diagram of Data Fusion SystemFeaturesSensors
TransformationFunctions
FusionFunctional
f1
f2
fi
fn
T11
T21
Ti1
Tn1
F[Tij(fk)]
FusionTechniques
• Heuristic
• Logic OR/AND• Averaging• Least Square• Geometric• Regularization• Bayesian• Possibility (Fuzzy)
T11(f1)
T21(f2)
Ti1(fi)
Tn1(fn)
FUSION
Electrical & Computer Engineering Department
The University of Tennessee, Knoxville334 Ferris Hall, Knoxville, TN, 37996-2100, USA 8
Core Technologies at the IRIS Lab.
3 - DATA VISUALIZATION
• Visualizing Reality
• Hardware, Object, View
Dependent Display
2 - SCENE DESCRIPTION
• Scene Segmentation• Multi - resolution Data Analysis• Data Reduction• Data Enhancement• Object Modeling
1 - SCENE BUILDING
• Data Acquisition and Sensor Characterization
• Sensor Placement• Data Registration• Data Fusion
Multi-Modal Model
Multi-Resolution,Multi-Modal,
Object Based Model
Multi-Modal Model
Electrical & Computer Engineering Department
The University of Tennessee, Knoxville334 Ferris Hall, Knoxville, TN, 37996-2100, USA 9
Univ. of Michigan
Univ. of Alaska
Clemson Univ.
Univ. of Iowa
Other Gov. Agencies: DOD (TACOM, Army NVL), DOE, FAA, etc.
ARMY/TACOM - UTENN-IRIS
Thrust Area 1: Intelligent Vehicle Dynamics & Control
UT IRISL A B O R A T O R Y
Thrust Area 3: High Performance Structures & Materials
Thrust Area 5: Integrated System Design & Simulation
Oakland Univ.
Wayne State Univ.
Univ. of Wisconsin
IndustryPartners
Thrust Area 2: Synthetic & Virtual Environments
Thrust Area 4: Advanced & Hybrid Powertrains
http://arc.engin.umich.edu/
Electrical & Computer Engineering Department
The University of Tennessee, Knoxville334 Ferris Hall, Knoxville, TN, 37996-2100, USA 10
Tasks
Flow diagram illustrating the five primary tasks that comprise the proposed research.
Flow diagram illustrating the five primary tasks that comprise the proposed research.
Platform Outfitting
Data Acquisition
Data ProcessingAnd Registration
Data FusionAnd Integration
Data Reduction
1
2
3
4
5Complete Scene Model Suitable For NADS
Preliminary LRI Database(no texture or attributes)
Dense Scene Database(too complex for real time)
http://www.nads-sc.uiowa.edu/
Electrical & Computer Engineering Department
The University of Tennessee, Knoxville334 Ferris Hall, Knoxville, TN, 37996-2100, USA 11
Multi-Sensor UTENN-IRIS Mobile Mapper
Vehicle
Control Room
Sensors
Communications and Sensors
Mast
Electrical & Computer Engineering Department
The University of Tennessee, Knoxville334 Ferris Hall, Knoxville, TN, 37996-2100, USA 12
Full 3D model recovered by integrating 4 views: 624,265 triangles
Riegl Laser Range Scanner:
• 9100x1100 image size (Max)•1-2 in standard deviation•Range, RGB color, Intensity
33--D Reconstruction from Real Range DataD Reconstruction from Real Range Data
• 70 (50 US + 20 Overseas) Faculty, Staff, and Students
• $3.5M of Externally Funded Research Per Year
• 40,000 square feet of Office and Laboratory Space
• Research AreasWarfighter Security & Physical SecurityNuclear SecurityAirport SecurityHarbor Security
• Major Funding AgenciesUS ArmyNational Nuclear Security AdministrationTSA – Department of Homeland SecurityOffice of Naval Research
ID CARD
1
UTK IRIS: Imaging, Robotics, and Intelligent Systems
2
3
4
Warfighter Security / Physical SecurityUS Army & Office of Naval Research
Nuclear SecurityNational Nuclear Security Administration
Airport SecurityTSA – Department of Homeland Security
Harbor SecurityOffice of Naval Research
IRIS Research Themes
5
• 3D Reconstruction Based on Stereo Images• Modeling Objects and Scenes from Multi-modal Imagery• Creating Wide-View Representations of Real-World Environments• Multiresolution Analysis of Irregular Meshes with Attributes• Reverse Engineering of Mechanical Parts• 3D Object Reconstruction Using Boolean Operations on Supershapes• Segmentation Based on Perceptive Metrics• Digitization of Real World Environments Using Vehicle-borne Scanning• Image Acquisition and 3D Modeling on Microscopic Scales
• Weapon Detection System for Under Vehicle Inspection• A Module-Based Generic Architecture for Mobile Robots• A Full-Information Sampling Strategy• Hardware/Software Development of 1D Laser Range Scanner• Simulation and Segmentation of Thermal IR Images• Informational Approach to Shape Description• Fusion of Visual and Thermal Face Recognition• A Combinatorial Face Detection Technique using Color• Co-registration of Thermal and Visual Image Pairs• 3D Face Recognition
Warfighter Security / Physical Security
On-Going Research Projects
Harbor SecurityAirport Security
Nuclear Security
• Image Fusion on Dual Energy X-Ray Images• Entropy-Based Multilevel Segmentation• Computer Simulation of Luggage• PTZ Camera Driven from Wide-Angle Camera• Multi-Object Motion Pattern Classification for Surveillance• Automatic Target Acquisition/Tracking Using Color and Shape• Illumination Invariant Video Tracking and Surveillance
• Data Acquisition and Transfer for Remote Video Surveillance• Hybrid 3D Modeling of Large Landscapes from Satellite Maps• Sensor Planning for Surveillance and Modeling• Multi Perspective Mosaicking of Harbor Vessels• Wireless Video and Control with PTZ Camera• Implementation of 3D Maps on Internet with Customizable Interface• Multiple Sensor Synchronization • Radar Ranging and Tracking
6
• Developing Shape Correlation methods for registering multiple rangeviews.
• Automatic alignment of color and rangdata.
• Rendering andmanipulation otextured mode
• Developing Shape Correlation methods for registering multiple rangeviews.
• Automatic alignment of color and rangdata.
• Rendering andmanipulation otextured mode
ans multi-modal dainput.
• Efficient 3-D surface model building, texturmapping and rendering.
multi-modal dainput.
• Efficient 3-D surface model building, texturmapping and rendering.
3D Scans of partsAligne
Registration
Path Determination from Multiple
Sensors
Path Determination from Multiple
Sensors
Geometry and Texture
Information
Geometry and Texture
Information
ctcoet
•SlafoAgstFl(FDtiocoet
Detection for Against Flow atExit Lane Simulation
intruder is calculated tocontrol PTZ camera for capturing froimages
• Real-time target initialization for color tracking
i
dy
dx
iy
thch
),( ji
ix
j
CamerahandoverCamera
handoverTarge
initializaTarge
initializa
Tracking uTracking u
resulting from different
lighting and orientation of the object.
e images and also the original color
AccomplishmentsSurvey of various color constancy an
llumination invariant tracking algorithmIllumination invariant Object
R iti i H hi Filt i
Example of a Macbeths chart under different lighting conditions
images.
Performance
visualized con• Application---
setup for humface scanning
minal-based control window
- from stereo images
video
Reconstructed Face
LASER SCANNED VIDEO INPUT
CALIBRATION PARAMETERS
LSC
VI
CAMERA CALIBRATION
3D Face Recognition System Diagram
Detectability
1
2
3
4
5
6
7
8
9
Ave
rage
Ran
k
Graphical User Interface for threat detection
Phase correlation and image segmentation
Results from Normal Vector Voting algorithshowing crease detection for a disc brake
ReconstructedModel
Positive Curvature
rface Registrati
● Real tivisualizon large scale models
● Simpliftion
● Frequencycontent assessme● Feature
detectiN i
solution levels
vel 3 Level 4
Initial noisy scanned mod
ange imageAccomplishments
•Manage real data ty(range images) creafrom scanners•Constructapproximating anddetail surfaces at ealevel of representatwith polynomial surfaces
Accomplishments•Manage real data ty(range images) creafrom scanners•Constructapproximating anddetail surfaces at ealevel of representatiwith polynomial surfaces
l
surface
at
level
Approximating surface
Control mesh
Detail coefficients
Image defines a control mesh to
The distance to the surface is
Wenewconpothadef
Input 3-D triangulated model
A distributor cap component
4 stroke engine
tic converterRepreswith m
e software to software to
inspectio• Develop
software interfacefor rangedata processinfor bettevisualizaon and hilevel processin
under vehscenes
Examplesrange da
nder vehicle 3D ange data cquisition & rocessing
eluction
Textured 3DModel
gnionh
he cart
Internet to transmit time video streams under remote PTZ controls
• Power and weathermanagement: Deveand deploy portableweather-hardened modules to carry allsystems component
Client1
ClientN
INTERNET
eless Surveillance System Architecture Rapid Prototype at IRIS Lab in
slice 4 slice 10 slice 21 slice 24 slice 41 slice 45 slice 50
A stack of 2images
results frothe one sc
The part is scanned under the microscope
The stack is processed to obtain a range image