Enhanced “GrabCut” Tool with Blob Analysis in Segmentation of Blooming Flower Images

Wooi-Nee Tan, Tejamaya Sunday, Yi-Fei Tan Faculty of Engineering, Multimedia University

Jalan Multimedia, 63100 Cyberjaya, Selangor, Malaysia 1wntan@mmu.edu.my

2tejamaya.sunday@yahoo.com 3yftan@mmu.edu.my

Abstract— This paper discusses the enhancement using blob analysis applied to automatic segmentation of “GrabCut” tool [1] for segmenting blooming flowers in color images. The automatic segmentation of “GrabCut” is used to initialize the segmentation, but the results are not effective and there is insufficient separation of foreground and background color distributions. In our proposed work, the segmented “GrabCut” image in RGB format is first converted to a binary image based on the V plane of the HSV color space. The morphology operators combining with set operations are then applied to fill up the holes of blob. This is then followed by blob filtering to eliminate the unwanted connected region. Finally, the segmented binary image is converted back to its RGB form. The proposed enhanced method achieves a more efficient extraction of blooming flower in a complex environment which cannot be trivially eliminated by the automatic segmentation of “GrabCut”. Keywords— image segmentation, GrabCut, blob analysis, morphology operators, set operations, blooming flower images

I. INTRODUCTION In image processing, one of the main tasks is image

segmentation which requires accurate cropping of important features and the deletion of unwanted background information. The importance of image segmentation is obvious as the unwanted information should not remain in the segmented image such that it would affect quantitatively the features to be extracted in image analysis. Our target application of the segmentation technique is to segment the blooming flowers from digital images. The proposed segmentation method would then be incorporated in an automated flower recognition model for further recognition purpose. One of the challenges posed in flowers image segmentation is that the images are captured in natural scene under uncontrolled environment involves variations in viewpoints, occlusions, scale, lighting, background etc. [2,3]. The separation of background and foreground in this class of images is not straightforward due to variation in terms of shape and colors.

In previous work of segmenting the flower images, Das et. al. [4] has proposed to use the color domain of flowers in segmenting flowers. In their work, the background color is taken from the periphery of the image. This technique fails whenever the flowers reach the image periphery as flower color will be mixed with the background color. Saitoh and Kaneko [5] proposed a segmentation method based on a

frontal flower image and a leaf image with a black sheet placed under the flower and leaf. This method though can segment the flower effectively from its black background but the approach is not convenient and not environmental friendly since the flowers need to be plucked and placed on a black sheet. Saitoh et. al.[6,7] later on proposed a method of extracting flowers based on “Intelligent Scissors”. A similar segmentation algorithm has also been developed by Hsu et. al. [8]. However, their method works well when the flower in the image is in focus and that the background is out of focus. Anyhow, the work by Hsu provides an interactive interface which allows the user to correct the wrongly detected tracing edge. Nilsback and Zisserman [3] proposed the segmentation scheme based on an iterative manner by first using the color model to separate the foreground and background. This is then followed by a generic shape model for segmenting the petal structure. The segmentation is produced by using a MRF cost function optimized using graph cut. The process of shape model fitting and color model learning is then iterated until convergence. Their proposed scheme is not suitable for fields of flowers.

In this paper, we propose to use automatic segmentation of “GrabCut” tool [1] to initialize the segmentation, and then improve the segmentation by using blob analysis. “GrabCut” is an iterative algorithm that combines statistics and graph cut in order to accomplish detailed 2-dimensional segmentation, and is originally developed at Microsoft Research Cambridge, UK [1]. “GrabCut” consists of two phases, with automatic segmentation in the first stage that only requires a user to draw a bounding box around the foreground. The second phase requires the user interaction by providing more inputs of foreground and background points to improve the quality of the segmented image.

In view of the application of our segmentation tool is to be incorporated in the automated flower recognition model with minimal user input, the automatic segmentation of “GrabCut” is chosen here as the initial segmentation. However, the output from “GrabCut” gives satisfactory results only when the pictures of flower are taken with monotone background, but for flowers images captured in natural environment, this technique unlikely guarantee a constant and adequate results in general cases. The resulted segmented output consists of background noise; see Figure 1 for samples of unsuccessful segmented outputs by using the automatic segmentation of

978-1-4799-0545-4/13/$31.00 ©2013 IEEE

“GrabCut” alone. Thus, it is important to enhance the automatic segmentation of “GrabCut” output for a more desirable result. We propose to improve the initial segmented results by introducing blob analysis. The advantage of the proposed work is that no user input is required and the whole process can be carried up automatically. The outcomes of this work provide an adequate segmentation result of blooming flower images. The enhanced segmentation technique does not restrict to one class of flowers, it also provides desired results for various form of flower shape, colors, different viewpoint, lighting or even when there is deformation of petals. This paper is organized as follows. In Section 2, we describe the proposed methodology. Section 3 provides the experimental results and conclusions are given in Section 4.

(a) Input images

(b) Segmented images produced by automatic segmentation of “GrabCut”

Fig. 1. Segmentation results on a few sample images. (a) Input images (b) Segmented images producedA by automatic segmentation of “GrabCut”

II. METHODOLOGY The proposed methodology is to enhance the segmented

images quality produced by automatic segmentation of “GrabCut”. Subsequent to the initial segmentation from “GrabCut”, most likely that the result generated for flower images taken in natural scene is beyond expectations due to existence of noise in the form of regions of con-necting pixels spreading out in some parts of the images. For flower images used as the input image for automated flower recognition model, the targeted object under interests is one whole blooming flower. The ideas we proposed here is to identify all the connected regions, determine the connected region which consists of the blooming flower and thus eliminate those unwanted region finally.

Let the original input image denote as O, and it is of size NM × . The segmented image obtained after automatic

segmentation of “GrabCut” is denoted as I. We observe that I consists of various connected regions with similarity in properties such as color and brightness compared to the surrounding neighborhood. These regions are known as blob in computer vision, and some of these blobs are the unwanted

region. The proposed steps in analyzing and filter the unwanted blob are as follows:

A. Step 1: Convert to HSV color space

In order to analyze the blob with similar brightness setting, image I is first converted to HSV color space which is natural to human perception in determining the color. The components in HSV color space are hue (H), saturation (S) and value (V) correspond to the common notation (θ,r,z) in cylindrical coordinate respectively. However, only the V plane is utilized here for further analysis since it provides the information of similarity in brightness setting for the blob under considerations. Here we denote the V plane of image I as IV. The values of V plane in image IV are in the range of [0,1]. B. Step 2: Convert to binary image

After conversion to V plane, the image is further converted to binary image by scaling the values of IV to either black with the value of 0 or white with the value of 1. A threshold value can be specified so that the pixels with the V plane values greater than the threshold value will be given the value 1 (white) and all other pixels will be replaced with value 0 (black). With this conversion, the image IV is separated to either background blob carrying the value of 0 or foreground blob with the value 1. We denote here the binary image obtained as IB.

C. Step 3: Perform binary morphological operation and set

operator In the generated binary image IB, there are likely to be

found some parts which are actually part of the targeted object but it is categorized as background. It is necessary to convert this region to pixels carrying values of 1 which is white. Otherwise, the system will not recognize it as part of the foreground blob. Therefore, we require to fill up the interior pixels in the foreground blob (the individual 0s pixels surrounded by 1s). The filling up process is performed based on dilation morphological operation, set complementation and set intersection operators defined as follows:

( ) ( ) NMkISXX CBkk ×=∩⊕= − ,...,3,2,1,1

with 0X is the initial binary image of size NM × with intensity values of 0 at pixel location ( )1,1 while all the remaining intensity values is 1, S is the 4-neighbours structuring element with values of 0, ( )C

BI is the complement of binary image IB. The process is an iterative process to be performed for NM × iterations. At the end of the operation, we obtain the filled binary image

BNMF IXI ∪= × , in which all the blobs appeared have been filled up. D. Step 4: Filter the unwanted blob

At this point, the image consists of several blobs. These blobs are then detected and labeled, and we denote the image with blob labeling as ID. Of course, the blob with the largest

area refers to the location of the targeted object which is the blooming flower. In order to filter all the unnecessary blobs, the area of each blob is calculated. Those blob with the area value cannot achieve a specified value will then be eliminated. We denote here the binary image after blob filtering as IS. E. Step 5: Restore the original color

The blob filtering process is completed at this point. Finally, the binary image IS with value of 1 is then restored to its original color by copying the RGB information in the input image O to the corresponding foreground pixels. We then successfully obtained the segmented image IC as desired.

III. RESULTS AND DISCUSSION In order to test the reliability of the proposed segmentation

process, experiments have been performed on several different types of commonly found flowers in Peninsular Malaysia. As a preliminary study, we restrict our scope to 5-petal flowers. Table 1 gives various images obtained in various steps during the process of the proposed blob analysis for 3 different types of flowers, namely ‘Alamanda’(A), ‘Turnera subu-lata’(TS) and ‘Jasminium sambac’(J). The threshold value setting in Step 2 is 0.5.

TABLE I

IMAGES PRODUCED IN VARIOUS STEP OF THE PROPOSED BLOB ANALYSIS

O I IV IB

A

TS

J

IF ID IS IC

A

TS

J

The output obtained after the automatic segmentation of

“GrabCut” process (image I) as shown in Table 1 is not

satisfactory with some noise existing. After the proposed enhancement process, the unwanted blob consists of the noise have been successfully eliminated and better segmentation results than using automatic segmentation of “GrabCut” alone is obtained. The segmentation process is further applied to more blooming flowers images taken in natural scene. More experiment results can be seen in Table 2.

The segmented images though successfully eliminate the unwanted noise produced by the automatic segmentation of “GrabCut”, however the edges of the blooming flower is not smooth, but these segmented results with blooming flower solely are sufficient to be input for the automated flower recognition model. If the smoothness of the flower edge is to be rectified, then the dilation morphology operator can be applied which is expected to provide more satisfactory results.

TABLE 2

EXPERIMENTAL RESULT FOR MORE INPUT IMAGES

Flower Type

Original Image

O

“GrabCut” Segmentation

I

Enhanced Segmentation

IC

‘Turnera subulata’

‘Adenuim’

‘Hibiscus’

‘Allamanda cathartica’

IV. CONCLUSIONS We proposed to enhance the automatic segmentation of

“GrabCut” with blob analysis in performing the automatic blooming flower image segmentation. The proposed technique is tested on various set of images for flowers commonly found in Malaysia. The results achieved complete success in filtering the unwanted noise produced by initial segmentation by the automatic segmentation of “GrabCut”. The advantage of the proposed enhancements is that the process is automatic with no requirement of user input. This fit to the purpose of the development of the segmentation tool which is then to be used for automated flower recognition

model, in which the model is expected to be an automatic process with minimal user input.

For future study, this proposed segmentation technique would be incorporated with the petal’s shape descriptor [8] for automated flower recognition model. One of the future works is to add the dilation morphology operator as the last phase of this segmentation technique to smoothen the flower edge. The proposed segmentation technique can also be applied in segmenting other form of images, for example the images with more complex targeted object such as animal, vehicle, building etc., this will then be considered as one of the future applications.

REFERENCES [1] Rother, C., Kolmogorov, V., Blake, V., ““GrabCut” - Interactive

Foreground Extraction using Iterated Graph Cuts,” ACM Transactions on Graphics, vol. 23, pp. 309-314, 2004.doi:10.1145/1015706.1015720

[2] Guru, D.S., Sharath Kumar, Y.H., Manjunath, S., “Textural Features in Flower Recognition,” Mathematical and Computer Modeling, 1030-1036, 2010. doi: 10.1016/ j.mcm.2010.11.032

[3] Nilsback, M.E., Zisserman. A., “Delving into the Whorl of Flower Segmentation,” in Proc. BMVC, vol. 1, pp.570-579, 2007.

[4] Das, M., Manmatha, R., Riseman, E.M., “Indexing Flower Patent Images Using Domain Knowledge,” IEEE Intelligent Systems and their Applications, vol. 14(5), pp. 24-33, 1999. doi:10.1109/5254.796084

[5] Saitoh, T., Kaneko, T., “Automatic Recognition of Wild Flowers.” in Proceedings 15th International Conference on Pattern Recognition, vol.2, pp.507-510, 2000. doi: 10.1109/ICPR.2000.906123

[6] Saitoh, T., Aoki, K., Kaneko, T., “Automatic Recognition of Blooming Flowers,” in Proceedings of the 17th International Conference on Pattern Recognition, vol. 1, pp. 27-30, 2004. doi:10.1109/ICPR.2004.1333997

[7] Hsu, T.Z., Lee, C.H., Chen, L.H., “An Interactive Flower Image Recognition System,” Multimedia Tools and Applications, vol. 53 (1), pp. 53-73, 2011. doi: 10.1007/s1042-010-0490-6

[8] Tan, W.N., Tan, Y..F., Koo, A.C., Lim, Y.P., “Petals’ Shape Descriptor for Blooming Flowers Recognition,” in Proc. SPIE 8334 - 4th International Conference on Digital Image Processing (ICDIP 2012), 83343K , 2012. doi:10.1117/12.96637