Team Nepenthes Subotica Tech - College of Applied Sciences, Subotica

Plant Recognition SystemPRS

Team members:Siniša VukovićAndor SíposAndor Nagl

Mentor:Zlatko Čović

1 | P a g e

Plant Recognition System

Content

2 | P a g e

s1. Introduction.....................................................................................................3

1.1 Team members........................................................................................3

1.2. Problem.......................................................................................................4

2. Solution...........................................................................................................53. System Architecture........................................................................................6

3.1. Administration Interface..............................................................................7

3.1.1. Plant Control Interface..........................................................................7

3.1.2. Neural Network Interface......................................................................8

3.1.3. Recognition Interface..........................................................................10

3.1.4. Model View ViewModel (MVVM).........................................................11

3.2. Database design.......................................................................................13

4. Windows Phone Application.........................................................................145. Recognition Theory.......................................................................................17

5.1. Image edge detection................................................................................17

5.2. Thinning....................................................................................................18

5.3. Leaf image token.......................................................................................18

5.4. Neuronal Network.....................................................................................19

6. Used Technologies.......................................................................................216.1. Microsoft Visual Studio..............................................................................21

6.2. Microsoft Expression Blend.......................................................................21

6.3. Windows Azure Platform...........................................................................21

6.4. Windows Phone 7.....................................................................................22

6.5. Windows Communication Foundation (WCF)...........................................22

6.6. Microsoft Silverlight...................................................................................22

6.7. MVVM Light Toolkit...................................................................................22

7. Conclusion....................................................................................................238. Future Plans.................................................................................................239. References...................................................................................................24

3 | P a g e

4 | P a g e

1. Introduction

1.1 Team members

Name: Siniša VukovićDate of birth: November 16, 1990.Place of birth: Subotica, SerbiaSchool: Subotica Tech

College of Applied Sciences,Subotica

Name: Andor SíposDate of birth: July 20, 1988.Place of birth: Subotica, SerbiaSchool: Subotica Tech

College of Applied Sciences,Subotica

Name: Andor NaglDate of birth: December 13, 1990.Place of birth: Subotica, SerbiaSchool: Subotica Tech

College of Applied Sciences,Subotica

2.

5 | P a g e

1.2. Problem

We live in a crazy, mixed up world with a lot of problems. There are some crucial problems that need to be solved right away, before it is too late. Unfortunately one of these problems is education.

Technology rapidly improves every day, which makes our lives much easier. Our generation was (and still is) part of this quick improvement, and we still remember what life was like before these high-tech gadgets and technologies. But that is not the case with the younger generations. This highly developed future will spoil them. They will get use to this luxury, and it will affect their whole life.

Unfortunately this is happening right now. People are wasting their precious time surfing the internet, playing games, and watching television all day. Surveys show that this is a real problem that is not to be ignored. Our opinion is the same and we cannot take this problem lightly!

The sad thing is, that this problem affects the young generations the most. A child’s mind is easily deceived by these things. This type of entertainment can be very dangerous, and it can also lead to addiction. Sadly we have seen it a lot of times with our own eyes.

As we said it earlier, this problem can have a negative impact on their lives. Unfortunately we will see that in their education as well.

6 | P a g e

2. Solution

Our aim is to make the learning world exciting and interesting for the new generations, and to take education itself to the next level.

With this goal we hope that we can grab the young ones’ attention, and show them the beauty of this world.

We know that this is a giant task, which cannot be solved with one idea or one project. This is one long road that we have to walk through together to achieve our goal. Our software is just a piece of the puzzle, but it is one step forward to victory.

INTERACTIVEEDUCATION

7 | P a g e

3. System Architecture

8 | P a g e

SQL Azure

Windows Phone ApplicationAdministration Interface

Windows Communication Foundation

Neural NetworkLINQ to SQL

Image 3-1: The System’s Architecture

3.1. Administration Interface

3.1.1. Plant Control Interface

In order to recognize a plant, we need to train the system. First we need to select the type of the plant and then load the images.

If it’s possible upload as many pictures as possible of the selected plant kind. Because system will be more effective, and there will be less chance of mistaking the type of the plant.

After every uploaded picture you will have to click on “Detect token” button. The uploaded image will only show its edges in vectors.

Image 3-2: Plant Control Interface

9 | P a g e

3.1.2. Neural Network Interface

Artificial Neural Networks (ANNs) are a new approach that follow a different way from traditional computing methods to solve problems. Since conventional computers use algorithmic approach, if the specific steps that the computer needs to follow are not known, the computer cannot solve the problem. That means, traditional computing methods can only solve the problems that we have already understood and knew how to solve. However, ANNs are, in some way, much more powerful because they can solve problems that we do not exactly know how to solve. That's why, of late, their usage is spreading over a wide range of area including, virus detection, robot control, intrusion detection systems, pattern (image, fingerprint, noise) recognition and so on.

ANNs have the ability to adapt, learn, generalize, cluster or organize data. There are many structures of ANNs including, Percepton, Adaline, Madaline, Kohonen, BackPropagation and many others. Probably, BackPropagation ANN is the most commonly used, as it is very simple to implement and effective. In this work, we will deal with BackPropagation ANNs.

BackPropagation ANNs contain one or more layers each of which are linked to the next layer. The first layer is called the "input layer" which meets the initial input (e.g. vectors of a leaf) and so does the last one "output layer" which usually holds the input's identifier. The layers between input and output layers are called "hidden layer(s)" which only propagate the previous layer's outputs to the next layer and (back) propagates the following layer's error to the previous layer. Actually, these are the main operations of training a BackPropagation ANN which follows a few steps.

A typical BackPropagation ANN is as depicted below. The black nodes (on the extreme left) are the initial inputs. Training such a network involves two phases. In the first phase, the inputs are propagated forward to compute the outputs for each output node. Then, each of these outputs are subtracted from its desired output, causing an error (an error for each output node). In the second phase, each of these output errors is passed backward and the weights are fixed. These two phases is continued until the sum of (square of output errors) reaches an acceptable value.

10 | P a g e

Image 3-4: Neural Network Interface

On Image 3-4 we can see the Neural network Interface. This is where we train and configure the Neural Network.

Based on the amount of images and network properties you normally need to specify around 500-1000 Training Cycles to get a good result in the recognition later. If the error rate drops below 0.01 you normally should encounter no problem in recognizing different leaf images.

The amount of input neurons for the network is normally twice the amount of tokens because of the sinus & cosine value for one token.

If the training is complete then we will see result on the left side of the interface.

11 | P a g e

3.1.3. Recognition Interface

Now that we trained our system to detect the plants, we can test it by uploading a picture of a leaf of the plant that we trained. If everything went well it will show us the name of the plant in the upper left corner.

Image 3-5: Recognition Interface

12 | P a g e

3.1.4. Model View ViewModel (MVVM)

The Model View ViewModel (MVVM) is an architectural pattern used in software engineering that originated from Microsoft as a specialization of the presentation model design pattern.

Image 3-6: MVVM Diagram

Elements of the MVVM pattern include:

Model: as in the classic MVC pattern, the model refers to either (a) a domain model which represents the real state content (an object-oriented approach), or (b) the data access layer that represents that content (a data-centric approach).

View: as in the classic MVC pattern, the view refers to all elements displayed by the GUI such as buttons, labels, and other controls

View model: the view model is a “model of the view” meaning it is an abstraction of the view that also serves in mediating between the view and the model which is the target of the view data bindings.

13 | P a g e

We chose the MVVM design pattern because we wanted to separate the user interface and the business logic. This is very useful because you can easily modify the program afterwards.

Image 3-7: ViewModel classes

14 | P a g e

3.2. Database design

The database is responsible for storage of all the information required for the system to run correctly. Microsoft SQL Azure was chosen as data storage, because it is easy to use and it is a free database management system.

The database diagram is shown on Image 3-5. There are a lot of tables in our database, but the most important ones are the following: The “Leaves” table and the tables that are in relationships with it, and the “NeuralNetwork” table.

Image 3-8: Database Design

15 | P a g e

4. Windows Phone Application

In this part of the documentation we will introduce the Plant Recognition System application for Windows Phone.

Image 4-1: PRS’s First Page

On Image 4-1 we can see the main page of the program. In the middle of the screen we can see a white camera inside a circle. If we press this button, we will activate a telephone’s camera view. Then we can take start taking pictures about the plants. It is important to put the leaves on a white paper, that way the

16 | P a g e

program can easily recognize the plant. You can see that in the “how to” option, which is located in the application bar.

Image 4-2: Plant’s Image Image 4-3: Plant’s Information

If we successfully took a picture of a plant, and the program recognized it, then we will see one picture of the captured plant from our database, and this page will also display the plant’s name, type and family (See Image 4-2).

If we scroll to the next page, it will contain many useful information about the captured plant. That way the user can learn a lot of interesting things about that plant (See Image 4-3).

17 | P a g e

Image 4-4: PRS’s Second Page

Image 4-4 is the second page of the application. This page will display the user’s latest photographed plants, with their names and types.

18 | P a g e

5. Recognition Theory

5.1. Image edge detection

On of the main tasks of this application is the detection of specific tokens in a leaf image. This tokens will then be the basis of the neuronal network calculations. Assuming that the image is a full 2D scan of a single leaf like in the examples below, we considered to use the well-known Prewitt Edge detection algorithm which we will explain further.

Prewitt edge detection produces an image where higher grey-level values indicate the presence of an edge between two objects. The Prewitt Edge Detection filter computes the root mean square of two 3x3 templates. It is one of the most popular 3x3 edge detection filters.The Prewitt edge detection filter uses these two 3x3 templates to calculate the gradient value:

-1 0 1 1 1 1 -1 0 1 0 0 0 -1 0 1 -1 -1 -1

X Y

Now consider the following 3x3 image window: +------------+ | a1 a2 a3 | | a4 a5 a6 | | a7 a8 a9 | +------------+

where:

a1 .. a9 - are the grey levels of each pixel in the filter window X = -1*a1 + 1*a3 - 1*a4 + 1*a6 - 1*a7 + 1*a9 Y = 1*a1 + 1*a2 + 1*a3 - 1*a7 - 1*a8 - 1*a9 Prewitt gradient = SQRT(X*X + Y*Y)

All pixels are filtered. In order to filter pixels located near the edge of an image, edge pixels values are replicated to give sufficient data.

19 | P a g e

5.2. Thinning

As discussed earlier, the idea of identifying a specific leaf image`s species here is that the outer frame of a leaf is enough to specify the species it belongs to. To accomplish that, it is necessary to identify this outer frame exactly. The previously applied Prewitt Edge detection normally just identify the edges with a preconfigured threshold and after this edge detection we have to perform a thinning algorithm to minimize this threshold-based edge to a one-line frame where we then can apply a sort of token recognition as discussed later.The used thinning algorithm here processed the image recursively and minimizes the found lines to a one-pixel wide one by comparing the actual pixel situation with specific patterns and then minimizes it.

5.3. Leaf image token

The central part of this application are the tokens of each leaf image that are found after the image processing is through with it. This runs in Windows Azure Cloud Platform. What exactly stands behind the idea of this tokens and how we defined this tokens should be explained here in detail.The idea behind the transfer of the leaf image shape into a neuronal network usable form is, that the cosine and sinus angles of the shape represents the criterias of a recognition pattern.

Image 5-1: Leaf Token

Image 5-1 shows a part of a leaf image that was already processed through the above mentioned edge detection and thinning algorithms.

20 | P a g e

To give you an idea of what you see in this image, here is a short list:

Green line: The shape of the leaf image after successful edge detection & thinning.

Red Square: This square represents a point on the shape of the leaf image from which we are going to draw a line to the next square.

Blue line: The compound of the center of two squares from which we are going to calculate the cosine and sinus angle. Such a blue line is a representation of a leaf token.

If you now take a deeper view on the small triangle zoom on this image you should recognize that it shows a right-angled triangle. This and the summary of all triangles of a leaf image are the representation of the tokens of a leaf from which we can start the neuronal network calculations.

Image 5-2: Triangle which represents the tokens

On Image 5-2 you see a small image of the right-angled triangle which represents a token of a single leaf image. Here it should be clear now that the angles A and B are the two necessary parts which will be fit into the neuronal network layers.

With these two angles we can exactly represent the direction of the hypotenuse from point P1 to P2 which is absolutely necessary for the representation of a leaf image.

5.4. Neuronal Network

Another main part of this work is the integration of a feed-forward Backpropagation neuronal network. As described earlier the inputs for this neuronal network are the individual tokens of a leaf image, and as a token normally consists of a cosine and sine angle, the amount of input layers for this network are the amount of tokens multiplied by two.

21 | P a g e

Image 5-3: Neural Network

The image on the right side should give you an idea of the neuronal network that takes place in the Leaves Recognition application.We have chosen a feed-forward Backpropagation network because it was part of the task to show that just a Backpropagation network and the shape of a leaf image is enough to specify the species of a leaf.The implemented network also just has one input, hidden and output layer to simplify and speed-up the calculations on that java implementation.

To fill the input neurons of the network, we use the previous calculated leaf tokens like discussed in section 2.3. The number of output neurons is normally specified by the amount of different species because we use a encoded form to specify the outputs.All other behaviours of the network is specified by the normal mathematical principals of a Backpropagation network. If you want to get an idea of how such a Backpropagation network works, please refer to deeper explanations about the Backpropagation algorithm.

22 | P a g e

6. Used Technologies

6.1. Microsoft Visual Studio

Microsoft Visual Studio is an integrated development environment. It is used to develop console and graphical user interface applications. Microsoft Visual Studio includes a code editor supporting IntelliSense as well as code refactoring. The integrated debugger works both as a source-level debugger and a machine-level debugger. Other built-in tools include a forms designer for building GUI applications, web designer, class designer, and database schema designer.

6.2. Microsoft Expression Blend

Expression Blend, Visual Studio, Silverlight and .NET provide the most compelling and seamless design and development workflow on the market today. Rapidly iterate on both the user experience and core architecture, evolving your ideas quickly from initial prototype through to completed project.

6.3. Windows Azure Platform

Microsoft Windows Azure Platform is a Microsoft cloud computing platform used to build, host and scale web applications through Microsoft data centers. Azure is classified as platform as a service and forms part of Microsoft's cloud computing strategy, along with its software as a service offering, Microsoft Online Services. The platform consists of various on-demand services hosted in Microsoft data centers and commoditized through three product brands.

23 | P a g e

6.4. Windows Phone 7

Windows Phone 7 is a mobile operating system developed by Microsoft, and is the successor to its Windows Mobile platform. Unlike its predecessor, it is primarily aimed at the consumer market rather than the enterprise market. Applications and games for Windows Phone 7 must be based on XNA or a WP7 specific version of Silverlight.

6.5. Windows Communication Foundation (WCF)

WCF is an application programming interface (API) in the .NET Framework for building connected, service-oriented applications. WCF is implemented primarily as a set of classes on top of the .NET Framework’s Common Language Runtime (CLR). This lets .NET developers build service-oriented applications in a familiar way. WCF allows creating clients that access services. Both the client and the service can run in pretty much any Windows process — WCF doesn’t define a required host. Wherever they run, clients and services can interact via SOAP, via a WCF-specific binary protocol, and in other ways.

6.6. Microsoft Silverlight

Microsoft Silverlight is an application framework for writing and running rich Internet applications, with features and purposes similar to those of Adobe Flash. The run-time environment for Silverlight is available as a plug-in for web browsers running under Microsoft Windows and Mac OS X. While early versions of Silverlight focused on streaming media, current versions support multimedia, graphics and animation, and give developers support for CLI languages and development tools. Silverlight is also one of the two application development platforms for Windows Phone.

6.7. MVVM Light Toolkit

The main purpose of the toolkit is to accelerate the creation and development of MVVM applications in WPF, Silverlight and in the Windows Phone.Like other MVVM implementations, the toolkit helps you to separate your View from your Model which creates applications that are cleaner and easier to maintain and extend. It also creates testable applications and allows you to have a much thinner user interface layer (which is more difficult to test automatically).

24 | P a g e

7. Conclusion

Plant Recognition System is an educational and entertaining software for everyone. Our particular targets are the younger generations, and the adventurers who love spending time in the great outdoors, but it can be interesting for everyone.

Like we said it earlier, our goal is to draw people’s attention towards education, to show its beauty, and to make education itself more entertaining, interactive and exciting for everyone.

We hope that our project can be a building block to this dream, or maybe it will inspire others to pursue our goal, and together we can make the world a better place.

8. Future Plans

Although the program is complete, there are many things that we can improve and add to it. We still have many useful and interesting ideas that we can use, like the following:

Upload the database with as many plants as possible. Improve the recognition algorithm. Develop the application for Windows 8 Metro.

25 | P a g e

9. References

[01] http://en.wikipedia.org/wiki/Neural_network

[02] http://en.wikipedia.org/wiki/Artificial_neural_network

[03] http://msdn.microsoft.com/

[04] http://edis.ifas.ufl.edu/

[05] http://www.codeproject.com/Articles/19323/Image-Recognition-with-NeuralNetworks

[06] http://flavia.sourceforge.net/

[07] http://galasoft.ch/mvvm/

26 | P a g e