49
Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering Clemson University { yinxial, stb }@clemson.edu

Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Embed Size (px)

Citation preview

Page 1: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot

Yinxiao Li and Stanley T. BirchfieldDepartment of Electrical and Computer Engineering

Clemson University{ yinxial, stb }@clemson.edu

Page 2: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

MotivationGoal: Segment the floor in a single corridor image

Why?• obstacle avoidance• mapping• autonomous exploration and navigation

Page 3: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

MotivationGoal: Segment the floor in a single corridor image

Why?• obstacle avoidance• mapping• autonomous exploration and navigation

Page 4: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Outline

• Previous Work

• Detecting Line Segments

• Score Model for Evaluating Line Segments

– Structure Score

– Bottom Score

– Homogeneous Score

• Experimental Results

• Conclusion

Page 5: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Outline

• Previous Work

• Detecting Line Segments

• Score Model for Evaluating Line Segments

– Structure Score

– Bottom Score

– Homogeneous Score

• Experimental Results

• Conclusion

Page 6: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Previous WorkFree space detection

• Combination of color and histogram (Lorigo 1997)

• Optical flow (Stoffler 2000, Santos-Victor 1995)

• Stereo matching (Sabe 2004)

Floor detection• Apply planar homographies to

optical flow vectors (Kim 2009, Zhou 2006)

• Stereo homographies (Fazl-Ersi 2009)

• Geometric reasoning (Lee 2009)

Limitations• Multiple images for motion• Multiple cameras for stereo• Require different colors for floor and wall• Computational efficiency• Calibrated cameras • Assume ceiling visible

Our contribution: Segment the floor in real time using a single image captured by a low-height mobile robot

Page 7: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Challenging problem: Reflections

Also:• variety of poses (vanishing point, ceiling not always visible)• sometimes wall and floor color are nearly the same

Page 8: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Outline

• Previous Work

• Detecting Line Segments

• Score Model for Evaluating Line Segments

– Structure Score

– Bottom Score

– Homogeneous Score

• Experimental Results

• Conclusion

Page 9: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Douglas-Peucker Algorithm

Purpose: Reduce the number of points in a curve (polyline)

Algorithm:

1. Connect farthest endpoints2. Repeat

1. Find maximum distance between the original curve and the simplified curve

2. Split curve at this point Until max distance is less than

threshold

David Douglas & Thomas Peucker, "Algorithms for the reduction of the number of points required to represent a digitized line or its caricature", The Canadian Cartographer 10(2), 112–122 (1973)

Page 10: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Douglas-Peucker Algorithm

Purpose: Reduce the number of points in a curve (polyline)

Algorithm:

1. Connect farthest endpoints2. Repeat

1. Find maximum distance between the original curve and the simplified curve

2. Split curve at this point Until max distance is less than

threshold

Page 11: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Douglas-Peucker Algorithm

Purpose: Reduce the number of points in a curve (polyline)

Algorithm:

1. Connect farthest endpoints2. Repeat

1. Find maximum distance between the original curve and the simplified curve

2. Split curve at this point Until max distance is less than

threshold

Page 12: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Douglas-Peucker Algorithm

Purpose: Reduce the number of points in a curve (polyline)

Algorithm:

1. Connect farthest endpoints2. Repeat

1. Find maximum distance between the original curve and the simplified curve

2. Split curve at this point Until max distance is less than

threshold

Page 13: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Douglas-Peucker Algorithm

Purpose: Reduce the number of points in a curve (polyline)

Algorithm:

1. Connect farthest endpoints2. Repeat

1. Find maximum distance between the original curve and the simplified curve

2. Split curve at this point Until max distance is less than

threshold

Page 14: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Douglas-Peucker Algorithm

Purpose: Reduce the number of points in a curve (polyline)

Algorithm:

1. Connect farthest endpoints2. Repeat

1. Find maximum distance between the original curve and the simplified curve

2. Split curve at this point Until max distance is less than

threshold

Page 15: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Modified Douglas-Peucker Algorithm

original algorithm: dallowed is constantmodified algorithm: dallowed is given by half-sigmoid function

dallowed

original modified

Page 16: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Detecting Line Segments (LS)

Page 17: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Detecting Line Segments (LS)

1. Compute Canny edges

Page 18: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Detecting Line Segments (LS)

1. Compute Canny edges

2. Modified Douglas-Peucker algorithm to detect line segments

Vertical LS: slope within ±5° of vertical direction Horizontal LS: Slope within ±45° of horizontal direction

dallowed

Page 19: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Detecting Line Segments (LS)

1. Compute Canny edges

2. Modified Douglas-Peucker algorithm to detect line segments

Vertical LS: slope within ±5° of vertical direction Horizontal LS: Slope within ±45° of horizontal direction

3. Pruning line segments (320x240) Vertical LS: minimum length 60 pixels Horizontal LS: minimum length 15 pixels Vanishing point (height selection): The vanishing point is computed as the mean of the intersection of pairs of non-vertical lines. It is used for throwing away horizontal line segments coming from windows, ceiling lights, etc.

dallowed

Page 20: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Outline

• Previous Work

• Detecting Line Segments

• Score Model for Evaluating Line Segments

– Structure Score

– Bottom Score

– Homogeneous Score

• Experimental Results

• Conclusion

Page 21: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Score Model

is assigned to each horizontal line

Structure Score Bottom Score Homogeneous Score

Weightshorizontal line

Page 22: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Score Model – Structure

Given a typical corridor image

1. |I(x,y)| > T1

(How to set threshold T1?)

|I(x,y)| > T1

Page 23: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Score Model – Structure

Given a typical corridor image

1. |I(x,y)| > T1 (gradient magnitude)

(How to set threshold T1?)

2. I(x,y) < T2 (image intensity)

(How to set threshold T2?)

|I(x,y)| > T1 and I(x,y) < T2

Page 24: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Score Model – Structure

Given a typical corridor image

1. |I(x,y)| > T1 (gradient magnitude)

(How to set threshold T1?)

2. I(x,y) < T2 (image intensity)

(How to set threshold T2?)

We applied SVM to 800 points in 200 images:

|I(x,y)| > T1 and I(x,y) < T2

Page 25: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Score Model – Structure

Given a typical corridor image

1. |I(x,y)| > T1 (gradient magnitude)

(How to set threshold T1?)

2. I(x,y) < T2 (image intensity)

(How to set threshold T2?)

Approximate using two thresholds:

|I(x,y)| > T1 and I(x,y) < T2

Page 26: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Score Model – Structure

Given a typical corridor image

1. |I(x,y)| > T1 (gradient magnitude)

2. I(x,y) < T2 (image intensity)

3. Now threshold original image using TLC (average gray level of pixels satisfying #1 and #2)

I(x,y) < TLC

Page 27: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Score Model – Structure

Given a typical corridor image

1. |I(x,y)| > T1 (gradient magnitude)

2. I(x,y) < T2 (image intensity)

3. Now threshold original image using TLC (average gray level of pixels satisfying #1 and #2)

4. Compute the chamfer distance between the line segments and structure blocks

Page 28: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Score Model – Structure

Ridler-Calvard Otsu

Our method

Our method is able to remove the spurious pixels on the floor caused by reflections or shadows

Method R-C Otsu Ours

Correctness 62% 66% 82%

Comparison of different threshold methods

Page 29: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Outline

• Previous Work

• Detecting Line Segments

• Score Model for Evaluating Line Segments

– Structure Score

– Bottom Score

– Homogeneous Score

• Experimental Results

• Conclusion

Page 30: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Score Model – Bottom

1. Connect the bottom points of consecutive vertical line segments to create “bottom” wall-floor boundary

Page 31: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Score Model – Bottom

1. Connect the bottom points of consecutive vertical line segments to create “bottom” wall-floor boundary

Page 32: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Score Model – Bottom

1. Connect the bottom points of consecutive vertical line segments to create “bottom” wall-floor boundary

2. Compute the distance of each horizontal line segment to the “bottom” wall-floor boundary

(red circled horizontal line segments indicates a positive contribution)

Page 33: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Outline

• Previous Work

• Detecting Line Segments

• Score Model for Evaluating Line Segments

– Structure Score

– Bottom Score

– Homogeneous Score

• Experimental Results

• Conclusion

Page 34: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Score Model – HomogeneousIdea: The floor tends to have larger

regions (due to decorations, posters, windows on the wall).

Algorithm:

1. Color-based segmentation of the image (using Felzenszwalb-Huttenlocher’s minimum spanning tree algorithm)

2. For each horizontal line segment, compute size of region

just belowsegment

size of largestsegment

Page 35: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Score Model – HomogeneousIdea: The floor tends to have larger

regions (due to decorations, posters, windows on the wall).

Algorithm:

1. Color-based segmentation of the image (using Felzenszwalb-Huttenlocher’s minimum spanning tree algorithm)

2. For each horizontal line segment, compute size of region

just belowsegment

size of largestsegment

Page 36: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Segmenting the floor

• Normalize the scores and sum• Threshold the final score:

• Connect the remaining line segments and extend the endpoints to the edges of the image

Page 37: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Outline

• Previous Work

• Detecting Line Segments

• Score Model for Evaluating Line Segments

– Structure Score

– Bottom Score

– Homogenous Score

• Experimental Results

• Conclusion

Page 38: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Evaluation Criterion• Wall- floor Boundary

– Green line: Detected wall-floor boundary

– Red line: Ground truth

Page 39: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Evaluation Criterion• Wall- floor Boundary

– Green line: Detected wall-floor boundary

– Red line: Ground truth

• Area– Blue shade area: misclassified area

– Red shade area: ground truth floor area

• Error Rate

• Segmentation is considered successful if rerr < 10%

%err

bluer

red

Page 40: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Sample Results89.1% success on database of 426 images:

Page 41: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Sample ResultsImages downloaded from the internet:

Page 42: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Sample ResultsSome successful results on failure examples from Lee et al. 2008

Original Image

Lee’s Ours

D. C. Lee, M. Hebert, and T. Kanade. “Geometric Reasoning for Single Image Structure Recovery”. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.

Page 43: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Sample Results• Failure examples

Checkered floor

Textured wall

Darkimage Bright

lights

Page 44: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Sample Results

• Video clip 1

Page 45: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Sample Results

• Video clip 2

Page 46: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Outline

• Previous Work

• Detecting Line Segments

• Score Model for Evaluating Line Segments

– Structure Score

– Bottom Score

– Homogeneous Score

• Experimental Results

• Conclusion

Page 47: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Conclusion• Summary

- Edge-based approach to segmenting floors in corridors

- Correctly handles specular reflections on floor- Nearly 90% of the corridor images in our database

can be correctly detected.- Speed: approximately 7 frames / sec

• Future work- Speed up the current algorithm- Improving score model by add more visual cues

(highly textured floors, low resolution, or dark environment)

- Use floor segmentation for mapping

Page 48: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Acknowledgement Clemson Computer Vision Group reviewers

• Zhichao Chen• Vidya Murali

Anonymous reviewers

Page 49: Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield Department of Electrical and Computer Engineering

Thanks!

Questions?

Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot