Content based image retrieval for agriculture crops

CONTENT-BASED IMAGE

RETRIEVAL FOR AGRICULTURAL

CROPS

Esraa M. Elhariri, MScScientific Research Group n Egypt (SRGE)

www.egyptscience.net

http://www.egyptscience.net/

Agenda

Introduction

Problem Definition

Scope of work & Proposed Solutions

Crops Ripeness

The Proposed Ripeness Classification System

Experimental Results

Conclusions & Future Work

Introduction

Product quality is one of the prime factors in

ensuring consistent marketing of crops.

Determining ripeness stages is a very

important issue in produce (fruits and

vegetables) industry for getting high quality.

Optimal harvest dates and prediction of

storage life are still mainly based on

subjective interpretation and practical

experience.

Introduction Cont…

Monitoring and controlling ripeness is a

very important issue in the fruit industry

since the state of ripeness during harvest,

storage and market distribution determines

the quality of the final product measured in

terms of customer satisfaction.

Problem Definition

Crops at greenhouse, storage containers are affected by

changes at temperature and humidity levels.

these changes can cause diseases, lost of nutritional value,

over-ripen and changes at texture, appearance and flavor and

this affect the quality of the agricultural crops.

Monitor the ripeness process of crops is very important issue

at agricultural and industry.

Problem Definition Cont…

So the human labors were required to visit the greenhouse and

check the humidity and temperature levels at specifics times

in according to plant type (e.g. strawberry need the

temperature level and humidity to be checked every 3 hours).

Doing this manually will be time consuming, cost high and

needs a lot of work and efforts.

Scope of work & Proposed

Solutions

This thesis concerns the study and development of different

approaches to help Labors via providing an automated

ripeness assessment system for agricultural crops.

Developing an automatic content based image retrieval

system for agriculture crops, which is tackling the problem

of ripeness assessment.

Having a comparative study among different classification

algorithms for the problem of ripeness classification.

Sweeter

Less Green

Softer

More palatable

Crops Ripeness

Choosing a crop

We choose Bell Pepper and Tomato for our research.

Different Ripeness Stages are described at the following

Charts.

The proposed classification

system: System Architecture

• Resizing images to 250x250 pixels for the followingreason:• Reducing their color index to avoid high

computation.

During this phase, the proposed approach prepares images

for the features extraction phase, The first step is :


system: Pre-processing Phase

• Remove gray shadow pixels.

• Convert image from RGB to Gray

image.

• Subtract image from background

image and compute a binary

mask.

• Apply the binary mask to original

image.

Background Removal Algorithm

Original Images

Images after background Removal

The second step is removing

image background to get region

of interest by using the following

procedure:



The third step is : converting each image from

RGB to HSV color space and extract HSV

components for the following reasons:

When applying classification using PCA and

Euclidean distance using both RGB and HSV color

features we got better results using HSV color.

From Survey, HSV color space as it is widely used

in the field of color vision and close to the

categories of human color perception

where R, G and B are colorcomponent of RGB colorspace



The pre-processing phase can

be summarized as the

following:

• Resize images to 250 X 250

pixels,

• Remove images background,

• Convert images from RGB color

space to HSV color space,

•Extract HSV components.



Since tomato surface color is the

most important characteristic to

observe ripeness of tomato, this

system uses color features for

classifying ripeness stages.

Two color descriptors which were

used here are color moments and

HSV colored histogram.


system: Feature Extraction Phase

Standard Equation:

The first descriptor is color

moments: we used mean, standard

deviation, and skewness which have

been proved to be efficient and

effective way for representing color

distribution in any image.

Mean, standard deviation and

skewness for a colored image of

size NxM pixels are defined by the

following equations:

Mean Equation:

Skewness Equation:



Colored histogram is the second color descriptor that shows representation of the

distribution of colors in an image. It represents the number of pixels that have

colors in each range of colors.

We used 1D 16x4x4 HSV colored histogram, with 16 level for hue and 4 level

for each of saturation and value.

Then we compute a combination 1D Features vector of the three color moments

and HSV histogram.



Principal component analysis is a

statistical common technique, which is

widely used in image recognition and

compression for a dimensionality

reduction, data representation and

features extraction tool as it ensures

better classification.

we use Principal Component

Analysis(PCA) for features vector

Transformation to extract good features

for the classification purpose.

IM1 F1 F2 : : : : : : Fk

IM2 : : : : : : : : :

::: : : : : : : : : :

IMn F1 F2 Fk

IM1 F1 : : Fm

IM2 : : : :

::: : : : :

IMn F1 : : Fm

PCA Algorithm

Whole Features

Extracted Sub-set of Features



During this phase, the proposed

approach uses Support Vector

Machine(SVM) as a classifier, this

phase is responsible for classify data

into 5 classes mentioned before.


system: Classification Phase

The Support Vector Machine (SVM) is a

Machine Learning (ML) algorithm that is

used for classification and regression of

high dimensional datasets with great

results. SVM solves the classification

problem via trying to find an optimal

separating hyperplane between classes.

SVM algorithm seeks to maximize the

margin around a hyperplane that separates

a positive class from a negative class.



Since SVM is a binary classifier, we have

used two different approaches to use it as

multi-class classifier:

One-against-All

One-against-One

Also we have tested our SVM classifier

using different kernel functions includes:

Linear kernel function,

Polynomial kernel function,

MLP kernel function,

RBF kernel function.



One against All:

1. Construct N binary SVM.

2. Each SVM separates one class

from the rest classes.

3. Train the ith SVM with all

training samples of the ith class

with positive labels, and

training samples of other classes

with negative labels.


system: Classification Phase (One-against-All

Approach)

At the begin, we have used One-

against-All multi-class SVM with

different kernel functions and total

of 230 images used for the

classification problem, it was

divided into 175 images as a

training dataset and 55 images as

a testing dataset.

Experimental Results:First Approach using One-against-All Approach

Training dataset is divided into 5 classes representing the different

stages of tomato ripeness.

Experimental Results:Training Dataset

Classification accuracy

using different kernel

function and different

sizes of training

classes:

Experimental Results:Results of First Approach

Experimental Results:Results of First Approach

From the previous figure we

found that classification

accuracy using SVM with

linear kernel function

increased by increasing

number of training images per

class.

Experimental Results: Accuracy of

linear Kernel Function using Different Training Size

Publications List

• Implementation of one-against-all multi-calss SVM for tomato

• Publication of Multiclass SVM Based Classification Approach for Tomato Ripeness

Submission : Jul 2013

Publication :Aug 2013

• Implementation of one-against-all & one-against-one multi-calssSVM

• Publication of Automated Ripeness Assessment System of Tomatoes Using PCA and SVM Techniques(book chapter)

Submission : Nov 2013

Publication: June 2014

• Implementation of one-against-one multi-calss SVM for bell pepper

• Publication of Bell Pepper Ripeness Classification based on Support Vector Machine

Submission : Dec 2013

Presentation: Apr 2014

• Implementation of different machine learning techniques for tomato as an extension of the first paper and the book chapter.

• Publication of Using machine learning techniques for evaluating tomato ripeness

Submission : Feb 2014

Publication: Oct 2014

• Comparative Study of different classification techniques for both crops.

• Publication of Random Forests Based Classification for Crops Ripeness Stages

Submission: Apr 2014

Publication : June 2014

Esraa Elhariri, Nashwa El-Bendary, Mohamed Mostafa M. Fouad, Jan Plato’s,

Aboul Ella Hassanien, Ahmed M. M. Hussein, “Multiclass SVM Based

Classification Approach for Tomato Ripeness.” In Proceedings of the Fourth

International Conference on Innovations in Bio-inspired Computing and

Applications(IBICA 2013), 22 – 24 August, Ostrava, Czech Republic, pp.175-186 ,

2013, Springer International Publishing.

Then, we have used both of

One-against-All & One-against-

One with 10-fold cross-

validation to generalize model

accuracy, using total of 250

images for both of training and

testing.

This figure shows how cross-

validation works.

TrainingTesting

Experimental Results: Second

Approach using both of OAO and OAA with Cross-Validation

One-against-One:

1-train k(k − 1)/2 binary SVMs

(1, 2), (1, 3), . . . , (1, k), (2, 3), (2,

4), . . . , (k − 1, k),

2-Select the one with the largest

vote.


system: Classification Phase (One-against-One

Approach)

Classification accuracy using different kernel function and total number of

250 image(used for both training and testing using 10 folds cross-

validation) using one against one approach for multi-class SVM

classification:

Experimental Results: One-against-

One(OAO) with 10 fold Cross-Validation Approach

Classification accuracy using different kernel function and total number of

250 image(used for both training and testing using 10 folds cross-

validation) using one against all approach for multi-class SVM

classification:


All(OAA) with 10 fold Cross-Validation Approach

The following figure show a comparison between the classification

accuracy of One-against-All(OAA) multi-class SVM and One-against-

One(OAO) multi-class SVM Approaches:

Experimental Results: Comparison

between of OAO & OAA Approaches

Publications List





















Esraa Elhariri, Nashwa El-Bendary, Aboul Ella Hassanien, Amr Badr:

"Automated Ripeness Assessment System of Tomatoes Using PCA and SVM

Techniques." Computer Vision and Image Processing in Intelligent Systems and

Multimedia Technologies. IGI Global, 2014. 101-130. Web. 9 Oct. 2014.

doi:10.4018/978-1-4666-6030-4.ch006.

Bell Pepper Ripeness

Assessment

Then we have worked on bell pepper:

we have used One-against-One

multi-class SVM with different

kernel functions and total of 175

images used for the classification

problem, they were used for both

training and testing dataset using

10 fold Cross-validation.

Experimental Results: One-against-One Approach

Training dataset is divided into 5 classes

representing the different stages of Bell Pepper

ripeness.

Experimental Results:Training Dataset


One (OAO) with 10 Cross-validation Approach

Classification accuracy using

different kernel function and

10-folds cross-validation.

Publications List





















Esraa Elhariri, Nashwa El-Bendary, Ahmed M. M. Hussein, Aboul Ella

Hassanien, and Amr Badr, “Bell Pepper Ripeness Classification based on Support

Vector Machine”, In Proceeding of IEEE ICET, 19 – 21 April, Cairo, Egypt.


system: System Architecture (Comparison

between SVM and LDA)

Then, we have used both of

One-against-All & One-against-

One, In addition to, LDA with

10-fold cross-validation to

generalize model accuracy,

using total of 250 images for

both of training and testing.

This figure shows how cross-

validation works.

TrainingTesting

Experimental Results: Second Approach

using both of OAO, OAA and LDA with Cross-Validation

Linear Discriminant Analysis

(LDA)

LDA is a commonly used

technique for data classification

and dimensionality reduction.

It’s basic idea is to find a linear

transformation that best

discriminate among classes, then

classification can be performed in

transformed space based on some

metrics(Euclidean distance)

Classification accuracy using different kernel function and total number of 250

image(used for both training and testing using 10 folds cross-validation) using one

against one approach for multi-class SVM classification:


One (OAO) with 10 Cross-validation Approach

Classification accuracy using different kernel function and total number of 250

image(used for both training and testing using 10 folds cross-validation) using one

against all approach for multi-class SVM classification:

Experimental Results: One-against-All

(OAA) with 10 Cross-validation Approach

Classification accuracy using total number of 250 image(used for both training

and testing using 10 folds cross-validation) using LDA classification:

Accuracy= 84 %

Experimental Results LDA with 10

Cross-validation Approach

The following figure show a comparison between the

classification accuracy of One-against-All(OAA) multi-class

SVM and One-against-One(OAO) multi-class SVM and LDA

Approaches:

Experimental Results: Comparison

between of OAO & OAA and LDA Approaches

Publications List













• Implementation of different machine learning techniques fortomato as an extension of the first paper and the book chapter.

• Publication of Using machine learning techniques for evaluatingtomato ripeness







Nashwa El-Bendary, Esraa El Hariri, Aboul Ella Hassanien, Amr Badr, “Using

machine learning techniques for evaluating tomato ripeness”, In International

Journal of Expert Systems with Applications, Volume 42, Issue 4, March 2015,

Pages 1892-1905, Elsevier, ISSN 0957-4174,

http://dx.doi.org/10.1016/j.eswa.2014.09.057.


system: System Architecture

Random

ForestsOAO-SVM

Experimental Results: One-against-One Approach

we have used One-against-One multi-

class SVM and Random Forests with

different kernel functions and total of 175

and 250 images for bell pepper and

tomato, respectively used for the

classification problem, they were used

for both training and testing dataset.



The Random Forests (RF) is one of the best

known classification and regression

techniques, which has the ability to classify

large dataset with excellent accuracy.

Random Forests algorithm generates an

ensemble of decision trees. Ensemble

methods main principle is to group weak

learners together to build a strong learner.



The input is entered at the top and as it traverses

down the tree, the original data is sampled in

random, but with replacement into smaller and

smaller sets.

The class of sample is determined using random

forests trees, which are of an arbitrary number.

Experimental Results: Training Dataset

For Bell Pepper, Training dataset is divided into 5 classes

representing the different stages of Bell Pepper ripeness.

Experimental Results: Training Dataset

For Tomato, Training dataset is divided into 5 classes

representing the different stages of Tomato ripeness.

Experimental Results: One-against-One

(OAO) & RF with 10 Cross-validation Approaches

Bell Pepper Ripeness

Classification accuracy using

Random Forest and SVM

using different kernel function

and 10-folds cross-validation.

Experimental Results: One-against-One

(OAO) & RF with 10 Cross-validation Approaches

Tomato Ripeness Classification

accuracy using Random Forest

and SVM using different kernel

function and 10-folds cross-

validation.

Publications List





















Esraa Elhariri, Nashwa El-Bendary, Aboul Ella Hassanien, Amr Badr, Ahmed M.

M. Hussein, Václav Snásel, “Random Forests Based Classification for Crops

Ripeness Stages”, In Proceedings of the Fifth International Conference on

Innovations in Bio-Inspired Computing and Applications (IBICA 2014), vol. 303,

23 – 25 June, Ostrava, Czech Republic, pp.205-215, 2014, Springer International

Publishing.


In this Thesis, a system for classifying the ripeness stages ofboth tomato and bell pepper has been developed.

For tomato ripeness assessment, the ripeness classificationaccuracy obtained by the OAO multi-class SVM approach isbetter than ripeness classification accuracy obtained by theOAA multi-class SVM and LDA approaches.

For bell pepper ripeness assessment, the ripeness classificationaccuracy obtained by the OAO multi-class SVM approach isbetter than ripeness classification accuracy obtained by theOAA multi-class SVM approach.


A number of future research could be achieved via classifyingdifferent objects or crops by involving other features (texture,shape, size, ... etc.) according to the classified objects nature.

Other Machine Learning approaches could be employed inorder to address the advantages and limitations of applyingeach of them.

Another direction of research is to use nondestructive/non-invasive detection technologies of food quality/maturity suchas hyperspectral imaging systems, colorimetric, Near InfraredSpectroscopy, and non-invasive smart sensing technologies

For further questions:Esraa M. El-hariri

[email protected]

Engineering

Content based image retrieval for agriculture crops