Upload
dhiraj-gurnani
View
126
Download
0
Embed Size (px)
Citation preview
AMAZON FARMERS FRESH
1 | P a g eCMPE 273 Group Project
Spring 2016
CMPE 273 – Enterprise Distributed Systems
FINAL GROUP REPORT
On
Amazon Market Fresh
Instructor:
Dr. Simon Shim
Submission Date: Submitted By:
May 1, 2016 Dhiraj Gurnani Satyateja Pothuru Ashwini Chellagurki Vimal Muraleedharan Nair Vaishampayan Reddy Pathuri
AMAZON FARMERS FRESH
2 | P a g eCMPE 273 Group Project
INDEX
1. Introduction: state your goals, purpose of system...................................32. System Design: Describe your chosen system design...............................33. Object Management Policy.......................................................................8
3.1 Planning and Requirement Analysis....................................................83.2 Consideration of the Functional Requirements..................................83.3 Development......................................................................................93.4 Testing the System...............................................................................9
4. Handling of Heavy Weight Resources........................................................94.1 Usage of Connection Pooling..............................................................94.2 Usage of Rabbit MQ............................................................................104.3 Usage of Redis Caching.......................................................................11
5. Policy used to decide when to write to data.............................................126. Implementation Details: Performance/Mocha Tests................................127. Use Cases/Work Flow – Screenshots…......................................................168. Dynamic Pricing Algorithm…......................................................................43
3 | P a g eCMPE 273 Group Project
AMAZON FARMERS FRESH
1. Introduction :
The goal of our system is to develop an application which is similar to AmazonFresh Farmers Market to demostrate the use of REST based Web Services in our Middleware technology , the use of RabbitMQ as a messaging platform for communication between front-end channels with backend systems, and implementing the connection pooling concept to effectively manage our connections and how it effects the overall performance of the system and finally to showcase as to where and which pages to use the SQL and NOSQL database in our system. Our Application offers wide range of grocery items for sale, as well as a subset of items for purchase to the customers.Items that are ordered are available for home delivery on the same day or the next day, depending on the time of the order and the availability of trucks which are also managed by our application.Here the local farmers can sell their fresh produce such as lettuce, tomato, potato, oranges, carrots, etc to customers by getting them approved by the admin. The farmers put their descriptions of products and introduction to their farms that includes history, farming methods, video tour, owner profile etc. Customers can browse many local farmers and items, and make purchases.They can check the order progress in the web and also check the real time locatin of the delivery truck through google maps.
2. System Design:
Block Diagram for Amazon Fresh
4 | P a g eCMPE 273 Group Project
AMAZON FARMERS FRESH
Schema Diagram:
Mongo DB:
AMAZON FARMERS FRESH
5 | P a g eCMPE 273 Group Project
Sessions (Implemented using Passport.JS and stored in Mongo)
System Design(Continued….)Amazon Fresh Client represents the front-end where customers can select various agricultural items from different forms. Functionality such as available fresh produce items, delivery method and description of farmers and forming methods can be accessed from this page. User interface is customized for the role of the logged in user. If the logged in user is a customer and if the request is made to buy a product then server enques this in a request queue with a specific correlation_id. This request will be forther forwarded to service defined for this type of request. It is here RabbitMQ plays
AMAZON FARMERS FRESH
6 | P a g eCMPE 273 Group Project
an important role of decoupling the different components to address scalability. After the request has been serviced, the response will be enqueued in the RabbitMQ response queue for the predefined correlation_id. Finally, it will be sent back to client for display. All the session level and configuration information is stored in the Redis cache. Once the user is authenticated, for the subsequent requests the information stored in the Redis cache will be retrived for the validation.
In our AmazonFresh Farmers Market application, we are making use of bootstrap, HTML5 and angularjs for the client side code and Node js for the server side code. The application provides the users with a AmazonFresh Farmers Market like interface with the basic functionalities like allowing the farmer to sell his fresh produce by creating his account and by updating his products which are in prior approved by the admin and are purchased by the customers who can view all the products, rate them and review them and are also provided with information about a registered delivery truck and can live track it while it is out for delivery.The customers are also provided with a bill at the end of each order placed.The products are also dynamically priced with their price varying on the days when they have high demand etc. The customer should be logged in while doing a checkout of the products , if not he can signup and carry out the request. The farmer,customer,admin product, trucks and billing information is stored in Mysql and is retrieved whenever required whereas the farmer introduction video and all images are stored in MongoDb for easy management.We have made use of connection pooling in the database to effectively manage our connections and to not create a overhead on the system as we will be testing our application on huge data. We have made use of RabbitMq as a messaging platform for high availability and to increase the performance.
Test Scenarios:
We have also used testing frameworks to test our application namely Jmeter for testing the performance of the system when subjected to varying load of the clients and the other namely mocha testing framework.
Github:We have used GitHub which is a web-based Git repository to host our Application.
URL: https://github.com/vaisham92/AmazonFreshMock
AMAZON FARMERS FRESH
7 | P a g eCMPE 273 Group Project
AMAZON FARMERS FRESH
8 | P a g eCMPE 273 Group Project
3.Object Management Policy
3.1 Planning and Requirement Analysis:
In this stage, we analyzed and performed requirement analysis, as per the project requirements in the three tiers which are as follows:
a) The client tier: The front-end users namely customers, farmers and the admin will interact with our applications and sends responses using the UI interface.
b) The middle tier/middleware: The majority of the processing of the inputs takes place in this tier, it performs the sending and receiving of the messages between distributed systems. It is at this most critical point that it needs to integrate new components or to scale existing ones as efficiently as possible.The modules have been implemented using RabbitMQ which implements the Advanced Message Queuing Protocol.
c) The third tier: This tier consists of database to store data of the system. At this tier, we also have constraints and relations that define the data. Most of the modules use MySQL as the relational database and the videos and images are stored using MongoDB as the NoSQL database to store data.
3.2 Consideration of the Functional Requirements:
As per the project requirement, we implemented different modules namely Farmers, Customers, Billing, Admin, Billing and Trucks. Each of the modules have an associated schema which are responsible for showing how object should be stored in database. Each objects exhibits their respective functionalities as per the project requirements.
3.3 Development:
According to the project requirement, we have created the UI and have implemented different functionalities. Additionaly, We have also implemented key features like maps using Google maps API and live tracking of the trucks by the customer track the exact location of the truck.We have used connection pool to efftectively manage our connections on huge load and Rabbit MQ to increase the sustainability while the load
AMAZON FARMERS FRESH
9 | P a g eCMPE 273 Group Project
in high.
AMAZON FARMERS FRESH
10 | P a g eCMPE 273 Group Project
3.4 Testing the System :
After the development of our application, on the basis of requirement, we have used testing frameworks to test our application namely Jmeter for testing the performance of the system when subjected to varying load of the clients and the other namely mocha testing framework.We have also drafted the test cases which were checked if all the requirement functionality is working fine or not.
4. Handling of Heavy Weight Resources
We have used the following to efficiently manage the heavy weight resources :
4.1 Usage of Connection Pooling:
Whenever an application needs an access to database it connects to the database. Hence, it will be disadvantageous if it connects every time a user sends a request for a particular data.so our application makes use of connection pooling where we maintain a pool of connection instances.Here we can create any number of connection objects and keep it in a pool so if a request comes it will take a instance from the pool and serve. After accessing the data and we no longer want the connection we release it back into the pool and it will be reused by some other function.In this way, it makes the access faster improves the performance of an application.
10 | P a g eCMPE 273 Group Project
AMAZON FARMERS FRESH
4.2 Usage of Rabbit MQ
Rabbit MQ is a message passing service which uses the AMQP protocol. It serves request from client to server and sends response from server to client. Messaging enables software applications to connect and scale. We have made separate queues for handling the data from different modules viz. admin, truck,farmers,product customer, billing etc. Applications can connect to each other, as components of a larger application, or to user devices and data. Messaging is asynchronous, decoupling applications by separating sending and receiving data.Hence by making use of RabbitMQ in our system we are making it reliable and highly available.
11 | P a g eCMPE 273 Group Project
AMAZON FARMERS FRESH
4.3 Usage of Redis Caching :
Redis is a remote data structure server and is extremely useful for cache management. Redis can be accessed by all the processes of your applications, possibly running on several nodes (something local memory cannot achieve). Redis memory storage is quite efficient, and done in a separate process. If the application runs on a platform whose memory is garbage collected (node.js, java, etc ...), it allows handling a much bigger memory cache/store. Redis can also persist the data on disk if needed. Redis can replicate its activity with a master/slave mechanism in order to implement high- availability. It allows for fine-grained control over eviction though a choice of six different eviction policies. Redis employs more sophisticated approaches to memory management and eviction candidate selection. Redis gives us much greater flexibility regarding the objects we can cache. Redis allows key names and values to be as large as 512MB each, and they are binary safe. Redis has six data types that enable more intelligent caching and manipulation of cached data, opening up a world of possibilities to the application developer.Hence By making use of Redis in our application we are making it highly available and efficient.
AMAZON FARMERS FRESH
12 | P a g eCMPE 273 Group Project
5. Policy used to decide when to write to data
In our application we have writen or have updated into our database only after the processing of our request is complete so that the data inserted into the database is correct and not incomplete.
6. IMPLEMENTATION DETAILS
1. Performance Testing Updates
Response Time
25
20
15
10
5
0100 200 300 400 500 1000
Number of Concurrent Users
Without Connection Pooling With Connection Pooling
Rabbit MQ + Connection Pool Rabbit MQ + Connection Pool + Redis
AMAZON FARMERS FRESH
13 | P a g eCMPE 273 Group Project
Note: Performed on heaviest call – getBillDetails
Num
ber o
f est
imat
ed tr
ansa
ction
s / se
c
Performance improvement Vs Concurrency
45%40%35%30%25%20%15%10%5%0%
100 200 300 400 500 1000
Concurrency
Connection PoolConnection Pool + Rabbit MQConnection Pool + Rabbit MQ + Redis Cache
AMAZON FARMERS FRESH
14 | P a g eCMPE 273 Group Project
2. Mocha Test Cases:
Pefo
rman
ce Im
prov
emen
t in
%
AMAZON FARMERS FRESH
15 | P a g eCMPE 273 Group Project
Source Code for Mocha test cases:
describe('http tests', function(){
it('Location Statistics API', function(done){ http.get('http://localhost:3000/api/admin/trips/locationStats',
function(res) {
})});
assert.equal(200, res.statusCode); done();
it('Revenue Statistics API', function(done){ http.get('http://localhost:3000/api/admin/trips/revenueStats',
function(res) {
})});
assert.equal(200, res.statusCode); done();
it('Get Passport Session', function(done){ http.get('http://localhost:3000/api/getSessionInfo', function(res) {
assert.equal(200, res.statusCode); done();
})});it('Get Pending Trips', function(done){
http.get('http://localhost:3000/api/admin/trips/getPendingTrips',function(res) {
})});
assert.equal(200, res.statusCode); done();
it('Get Bills', function(done){ http.get('http://localhost:3000/api/admin/trips/getBills', function(res) {
assert.equal(200, res.statusCode); done();
})});
AMAZON FARMERS FRESH
16 | P a g eCMPE 273 Group Project
});
7. Use Cases/UI Flows Screenshots. Welcome Page
Click Signin
AMAZON FARMERS FRESH
17 | P a g eCMPE 273 Group Project
Register User (Customer and farmer)
Validating the fields in Sign up page
AMAZON FARMERS FRESH
18 | P a g eCMPE 273 Group Project
AMAZON FARMERS FRESH
19 | P a g eCMPE 273 Group Project
Password Encryption(AES Algorithm)
Now Approve the farmer through Admin Page.
20 | P a g eCMPE 273 Group Project
AMAZON FARMERS FRESH
Enter Admin Details
Logging in as Admin
21 | P a g eCMPE 273 Group Project
AMAZON FARMERS FRESH
Go to Farmer Approval Page
Approve the Farmer
AMAZON FARMERS FRESH
22 | P a g eCMPE 273 Group Project
Approved and Logout
Admin Tracking
AMAZON FARMERS FRESH
23 | P a g eCMPE 273 Group Project
Now checking the Login Functionality
AMAZON FARMERS FRESH
24 | P a g eCMPE 273 Group Project
View the Farmer Welcome Page & click on edit farmer profile
AMAZON FARMERS FRESH
25 | P a g eCMPE 273 Group Project
Updated farmer profile
Click new product in homepage and add new product(Apple)
AMAZON FARMERS FRESH
26 | P a g eCMPE 273 Group Project
The new product created
Logging in as admin
AMAZON FARMERS FRESH
27 | P a g eCMPE 273 Group Project
Creating trips
Checking pending trips
AMAZON FARMERS FRESH
28 | P a g eCMPE 273 Group Project
Checking trip statistics
Checking Revenue statistics
AMAZON FARMERS FRESH
29 | P a g eCMPE 273 Group Project
Checking all bills placed
Approving a farmer
30 | P a g eCMPE 273 Group Project
AMAZON FARMERS FRESH
Approving a product
Viewing dynamic pricing
31 | P a g eCMPE 273 Group Project
AMAZON FARMERS FRESH
Entering the price with reference to dynamic price & updating
Signing in with customer
AMAZON FARMERS FRESH
32 | P a g eCMPE 273 Group Project
Entering zipcodes
Editing Customer profile
Click Edit
AMAZON FARMERS FRESH
33 | P a g eCMPE 273 Group Project
Home page part-1
AMAZON FARMERS FRESH
34 | P a g eCMPE 273 Group Project
Part-2
Clicking the fruits sub category
AMAZON FARMERS FRESH
35 | P a g eCMPE 273 Group Project
Clicking the apples product category
Clicking the specific apple (Product)
Searching a product
AMAZON FARMERS FRESH
36 | P a g eCMPE 273 Group Project
Farmer Products
AMAZON FARMERS FRESH
37 | P a g eCMPE 273 Group Project
Products Information Page with ratings and review
AMAZON FARMERS FRESH
38 | P a g eCMPE 273 Group Project
Inserting Rating (1 Star, 2 Star, 3 Star, 4 Star, 5 Star)
AMAZON FARMERS FRESH
39 | P a g eCMPE 273 Group Project
40 | P a g eCMPE 273 Group Project
AMAZON FARMERS FRESH
Adding Reviews and ratings
41 | P a g eCMPE 273 Group Project
AMAZON FARMERS FRESH
Showing average ratings
AMAZON FARMERS FRESH
42 | P a g eCMPE 273 Group Project
Cart Information
Getting Farmer Details on the Cart page.
AMAZON FARMERS FRESH
43 | P a g eCMPE 273 Group Project
Looking at the reviews in the product page
AMAZON FARMERS FRESH
44 | P a g eCMPE 273 Group Project
Cart page after adding product to cart
Clicki proceed to checkout and filling shipping details
AMAZON FARMERS FRESH
45 | P a g eCMPE 273 Group Project
Clicking continue and going to payment info page
Click pay now and going to confirmation page
AMAZON FARMERS FRESH
46 | P a g eCMPE 273 Group Project
Clicking profile button
Profile view
AMAZON FARMERS FRESH
47 | P a g eCMPE 273 Group Project
Clicking all placed orders
View placed orders
AMAZON FARMERS FRESH
48 | P a g eCMPE 273 Group Project
Signing out
Back to login page
AMAZON FARMERS FRESH
49 | P a g eCMPE 273 Group Project
8. Dynamic Pricing Algorithm.
Compute the average price of the same product for all quantity in the Data base. Compute the average price of the same product sold by the Amazon Fresh. Then find the average of step1 and step2 in the database which will shown as
the dynamic price based on demand supply model. If no similar products are available, DPA will show the result as null. The admin has an option to enter the price from the DFA Algorithm.
Implementation Example:
select sum(quantity) as sumOfQuantity from Products where category_id = 1 and subcategory_id = 1;
select quantity, price from Products where category_id = 1 and subcategory_id = 1 order by price desc;
select price from Products where category_id = 1 and subcategory_id = 1;
select max(price) from Products where category_id = 1 and subcategory_id = 1;
UI Implementation and How to apply the same:
50 | P a g eCMPE 273 Group Project
AMAZON FARMERS FRESH
51 | P a g eCMPE 273 Group Project
AMAZON FARMERS FRESH