Upload
trinhthuan
View
224
Download
4
Embed Size (px)
Citation preview
Object Recognition, Classification and Evaluation with PCLNovember 6, 2011
Outline
1. Introduction
2. Object Recognition with PCL
3. Training data
4. Recognition & Pose
5. Databases & Evaluation
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Object RecognitionCan we use PCL for object recognition?
filters io
features
octree search segmentation sample_consensus
surface
visualization
keypoints
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Object RecognitionCan we use PCL for object recognition? Yes, we can
filters io
features
search segmentation sample_consensus
surface
visualization
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Object RecognitionBuilding a PCL based object recognition app
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Segmentationpcl::apps::DominantPlaneSegmentation
Segmentation of a Kinect scene
typedef pcl::PointCloud<OpenNIFrameSource::PointT> Cloud;std::vector<typename Clout::Ptr>, ... > clusters;frame = camera.snap();
pcl::apps::DominantPlaneSegmentation<OpenNIFrameSource::PointT> dps;dps.setInputCloud (frame);dps.setMaxZBounds (1.0);dps.setObjectMinHeight (0.005);dps.setMinClusterSize (1000);dps.setWSize(3);dps.setDistanceBetweenClusters(0.1);dps.setDownsamplingSize (0.02);dps.compute_fast (clusters);
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Object RecognitionWhat else do we need? Descriptors
I Encode whole object with descriptor (globally)I VFHI CVFHI SHOTI SpinImageI ShapeContextI ShapeDistributionsI ...
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
VFHViewpoint Feature Histogram
I Encodes the surface of the object and the viewpointI Relative to the centroid and the average of the normals
I Efficient to compute
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
FeaturesVFH Estimation in PCL
1 using namespace pcl;2 typedef PointCloud<PointXYZ> CloudXYZ;3 typedef PointCloud<Normal> CloudNormal;4 typedef PointCloud<VFHSignature308> VFHSig;5 typedef CloudXYZ::Ptr CloudXYZPtr;6 typedef CloudNormal::Ptr CloudNormalPtr;78 CloudXYZPtr cloud(new CloudXYZ());9 CloudNormalPtr normals(new CloudNormal());10 // Compute normals11 ...1213 // Compute VFH14 VFHEstimation<PointXYZ, Normal, VFHSignature308> vfh;15 vfh.setInputCloud (cloud);16 vfh.setInputNormals (normals);17 KdTreeFLANN<PointXYZ>::Ptr tree (new KdTreeFLANN<PointXYZ> ());18 vfh.setSearchMethod (tree);19 VFHSig::Ptr vfhs (new VFHSig ());20 // Compute the features21 vfh.compute (*vfhs);
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Training dataWhat else do we need? Training data
I Given partial views from the KinectI we also want views for training
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Training dataSources:
I Capturing views with real sensors
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Training dataSources:
I Scanned object models
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Training dataSources:
I CAD models
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Using CAD modelsWhy and How
I Training on 3D models is ...
I easy: all you need is a CAD model and PCLI fast: some seconds / modelI complete: no missed views, viewpoints evenly spaced
around the objectI parameterizable: easily re-trainable with different
parametersI number of viewsI resolutionI noise levelI distance to object
I sensor independent: adapt code to simulate your favoritesensor characteristics (fov, aliasing, noise, errors, holes )
I providing additional information: entropy, aspect graphs,common reference frame
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Using CAD modelsWhy and How
I Training on 3D models is ...I easy: all you need is a CAD model and PCL
I fast: some seconds / modelI complete: no missed views, viewpoints evenly spaced
around the objectI parameterizable: easily re-trainable with different
parametersI number of viewsI resolutionI noise levelI distance to object
I sensor independent: adapt code to simulate your favoritesensor characteristics (fov, aliasing, noise, errors, holes )
I providing additional information: entropy, aspect graphs,common reference frame
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Using CAD modelsWhy and How
I Training on 3D models is ...I easy: all you need is a CAD model and PCLI fast: some seconds / model
I complete: no missed views, viewpoints evenly spacedaround the object
I parameterizable: easily re-trainable with differentparameters
I number of viewsI resolutionI noise levelI distance to object
I sensor independent: adapt code to simulate your favoritesensor characteristics (fov, aliasing, noise, errors, holes )
I providing additional information: entropy, aspect graphs,common reference frame
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Using CAD modelsWhy and How
I Training on 3D models is ...I easy: all you need is a CAD model and PCLI fast: some seconds / modelI complete: no missed views, viewpoints evenly spaced
around the object
I parameterizable: easily re-trainable with differentparameters
I number of viewsI resolutionI noise levelI distance to object
I sensor independent: adapt code to simulate your favoritesensor characteristics (fov, aliasing, noise, errors, holes )
I providing additional information: entropy, aspect graphs,common reference frame
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Using CAD modelsWhy and How
I Training on 3D models is ...I easy: all you need is a CAD model and PCLI fast: some seconds / modelI complete: no missed views, viewpoints evenly spaced
around the objectI parameterizable: easily re-trainable with different
parametersI number of viewsI resolutionI noise levelI distance to object
I sensor independent: adapt code to simulate your favoritesensor characteristics (fov, aliasing, noise, errors, holes )
I providing additional information: entropy, aspect graphs,common reference frame
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Using CAD modelsWhy and How
I Training on 3D models is ...I easy: all you need is a CAD model and PCLI fast: some seconds / modelI complete: no missed views, viewpoints evenly spaced
around the objectI parameterizable: easily re-trainable with different
parametersI number of viewsI resolutionI noise levelI distance to object
I sensor independent: adapt code to simulate your favoritesensor characteristics (fov, aliasing, noise, errors, holes )
I providing additional information: entropy, aspect graphs,common reference frame
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Using CAD modelsWhy and How
I Training on 3D models is ...I easy: all you need is a CAD model and PCLI fast: some seconds / modelI complete: no missed views, viewpoints evenly spaced
around the objectI parameterizable: easily re-trainable with different
parametersI number of viewsI resolutionI noise levelI distance to object
I sensor independent: adapt code to simulate your favoritesensor characteristics (fov, aliasing, noise, errors, holes )
I providing additional information: entropy, aspect graphs,common reference frame
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Using CAD modelsWhy and How
Obtain synthetic partial views of CAD models with PCL
std::string PLYModel = std::string(argv[1]);float resx = atof(argv[2]);float resy = resx;typedef pcl::PointCloud<pcl::PointXYZ> Cloud;std::vector <Cloud, ... > views_xyz;std::vector < Eigen::Matrix4f, ... > poses;std::vector<float> entropies;
pcl::visualization::PCLVisualizer vis;vis.addModelFromPLYFile (PLYModel, "mesh1", 0);vis.setRepresentationToSurfaceForAllActors ();vis.renderViewTesselatedSphere (resx, resy, views_xyz,
poses, entropies);
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Object Recognition
I What we have so farI Input from KinectI SegmentationI DescriptorsI Training data
I Matching against DB: L1,L2, Chi-Square,... with FLANNI best matching objects+views
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Object Recognition
I What we have so farI Input from KinectI SegmentationI DescriptorsI Training dataI Matching against DB: L1,L2, Chi-Square,... with FLANNI best matching objects+views
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Object RecognitionTime for a live-demo...
I Recognition/Classification Demo
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Object Recognition with PCLHow much code is to write?
pcl::visualization::PCLVisualizer vis("kinect");pcl::apps::DominantPlaneSegmentation<OpenNIFrameSource::PointT> dps;while(camera.isActive()) {frame = camera.snap();dps.setInputCloud (frame);dps.setMaxZBounds (Z_DIST_);dps.setObjectMinHeight (0.005);dps.setMinClusterSize (1000);dps.setWSize(3);dps.setDistanceBetweenClusters(0.1);dps.setDownsamplingSize (0.02);dps.compute_fast (clusters);vis.addPointCloud<OpenNIFrameSource::PointT>(filtered,"frame");for(size_t i=0; i < clusters.size(); i++) {vis.addPointCloud<OpenNIFrameSource::PointT> (clusters[i], random_handler, cluster_name.str());
recognizer.recognize (*cluster[i], model_ids, poses, confidences);
std::string category = getCategory(model_ids[0]);Eigen::Vector4f centroid;pcl::compute3DCentroid(*cluster[i], centroid);vis.addText3D(category, centroid, text_scale, 1, 0, 1, cluster_name.str(), 0);
}vis.spinOnce();
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Object Recognitionits even easier
I some generic classes do most of the work
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Object Recognition with PCLRecognition + Pose
I Aligned CAD models with objects
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Object Recognition with PCLRecognition + Pose
I classes for
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Object RecognitionTime for a live-demo...
I Recognition/Classification Demo with Pose
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
DatabasesWhere to get training data?
I Are there freely available sources out there?
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
DatabasesWhere to get data...
I Training data is freely availableI 3D-Net ( 3d-net.org )I Google’s 3D Warehouse, www.123dapp.com, turbosquid, ...I RoboEarthI Princeton Shape BenchmarkI Willow Garage household objects DBI KIT household object database, TUM, ...I ...
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
DatabasesHaving lots of CAD models...
I Nearest neighbor classification video
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
Evaluation
I Are there test databases out there?
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
EvaluationTest databases
I from 3D-NetI 1600 Kinect scenes, 10 Classes, single object, multiple
posesI +1600 scenes for 40 additional classes
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
EvaluationTest databases
I from SHRECI http://www.aimatshape.net/event/SHREC/I Shape Retrieval Contest of Range ScansI Range images captured using a Minolta Laser ScannerI segmented objects, as mesh
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
EvaluationTest databases
I from Lai’s RGBD-DBI http://www.cs.washington.edu/rgbd-dataset/I 300 common household objects, 51 classesI on Turntable with Kinect, 3 diff. viewing anglesI segmented objects, as colored point cloud
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
EvaluationEvaluation code
I e.g. to test your new feature/descriptorI implement your feature/descriptor in PCLI get training models, put into directoryI get a test databaseI use provided evaluation code
./category_evaluation-models_dir ../ICCV/ModelDatabase-training_dir ../ICCV/ModelDatabase_trained-test_dir ../3DNET/Cat10_TestDatabase-descriptor_name vfh-n_nearest_neighbours 4
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
EvaluationEasy evaluation
appl
e
bana
na
bottl
e
bow
l
car
donu
t
ham
mer
mug
tetr
a pa
k
toile
t pap
er
toilet paper
tetra pak
mug
hammer
donut
car
bowl
bottle
banana
apple
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1 2 3 4 5 6 7 8 9 10# Nearest Neighbours
0.0
0.2
0.4
0.6
0.8
1.0
Clas
sific
atio
n Ra
te
97% | 100% apple 82% | 92% banana 86% | 94% bottle 69% | 82% bowl 94% | 98% car 82% | 98% donut 100% | 100% hammer 99% | 100% mug 71% | 89% tetra pak 24% | 73% toilet paper 80% | 93% OVERALL
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation
The end
I PCL demo-apps and code to simplify evaluation andtesting
I helps you concentrate on your research area
Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation