A GraphChi Cluster

MTP Report submitted to

Indian Institute of Technology Mandi

for the award of the degree

of

B. Tech

by

Khushpreet Singh(B11016)

under the guidance of

Dr. Arti Kashyap

SCHOOL OF COMPUTING AND ELECTRICAL ENGINEERING

INDIAN INSTITUTE OF TECHNOLOGY MANDI

JUNE 2015

CERTIFICATE OF APPROVAL

Certified that the thesis entitled A GraphChi CLuster, submitted by Khushpreet Singh ,

to the Indian Institute of Technology Mandi, for the award of the degree of B. Tech has been

accepted after examination held today.

Date :

Mandi, 175001Faculty Advisor

CERTIFICATE

This is to certify that the thesis titled A GraphChi Cluster, submitted by Khushpreet

Singh, to the Indian Institute of Technolog, Mandi, is a record of bonafide work under my

(our) supervision and is worthy of consideration for the award of the degree of B. Tech of

the Institute.

Date :

Mandi, 175001Dr. Arti Kashyap

DECLARATION BY THE STUDENT

This is to certify that the thesis titled A GraphChi Cluster, submitted by me to the Indian

Institute of Technology Mandi for the award of the degree of B. Tech is a bonafide record

of work carried out by me under the supervision of Dr. Arti Kashyap. The contents of this

MTP, in full or in parts, have not been submitted to any other Institute or University for the

award of any degree or diploma.

Date :

Mandi, 175001Khushpreet Singh

Acknowledgments

Foremost, I would like to express my sincere gratitude to my project advisor/mentor Dr. Arti

Kashyap for the continuous support of our project and research, for her patience, motivation

and enthusiasm. Her guidance helped me in all the time of Major project and writing of this

thesis.

Besides my advisor, I would like to thank the faculty advisor: Dr. Dileep A. D. and rest

of the evaluation commitee: —– for their encouragement, insightful comments, and hard

questions.

I thank my fellow team partners for this project: Fatehjeet Sra, Ritish Rana for the

stimulating discussions, for the sleepless nights we were working together before deadlines,

and for all the fun we have had during our project. I could never imagine better partners than

you two for any project.

Khushpreet Singh

i

Abstract

Graph computation is a hot topic in research for past few years. With incresing data day by

day we need a faster and efficient paradign to process this amount of information. It will be

useful for lab experiments, online computation analyzing social network data. GraphChi is

a disk based graph computation system that is written in a vertex centric model for asyn-

chronous processing. It used parallel sliding window algorithm to load data into main mem-

ory and perform the queries that user has entered. Disk based approach makes graphchi

faster than commercialy used clusters with more number of machine used for graph com-

putaion. By deploying GraphChi and run it in a parallel fashion on a cluster might improve

its performance than used on a single machine. We used raspberry pis, low cost small com-

puting devices in order to configure cluster and wrote distributed queries for new system.

Various approaches, challanges, outputs and conclusion are discussed in the later chapter on

the thesis.

Keywords: GraphChi, Graph computation, Cluster, Raspberry pi, Graph partitioning

ii

Contents

Abstract ii

Abbreviations v

List of Symbols vi

List of Tables vii

List of Figures 1

1 Introduction 2

1.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Background and Related Work 5

2.1 Graph partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.1 PowerGraph . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.1.2 Edge partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2 Neo4j . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.3 GraphChi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.3.1 Features of GraphChi . . . . . . . . . . . . . . . . . . . . . . . . 8

2.4 Working of Graphchi . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.4.1 Parallel Sliding Window . . . . . . . . . . . . . . . . . . . . . . 9

2.4.1.1 Loads the graph from disk . . . . . . . . . . . . . . . . 9

2.4.1.2 Updates the vertices and edges . . . . . . . . . . . . . 9

iii

2.4.1.3 Writes updated values to disk . . . . . . . . . . . . . . 10

3 Implementation and Working 11

3.1 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.2 Graph Partition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.2.1 Sub-Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.3 Multi-Hop query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.4 Cluster of GraphChi Nodes . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.5 Raspberry Pi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4 Experimental Studies and Results 16

4.1 Algorithm Complexities . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4.1.1 For single Machine . . . . . . . . . . . . . . . . . . . . . . . . . 17

4.1.2 For Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4.2 Cluster Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4.2.1 Subsequent Query . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.2.2 Query Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.2.3 Distribution Time . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4.3 Raspberry Pi Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

5 Conclusion and Future Work 21

References 22

iv

Abbreviations

PSW - Parallel Sliding window

FTP - file transfer protocol

OS - Operating System

RAM - Random Access Memory

SSD - Secondary Storage Device

CMU - Carnegie Mellon University

SQL - Standard Query Language

I/O - Input/Output

v

Symbols

O - Upper Bound

vi

List of Tables

2.1 Runtime for different data sets for different query apps . . . . . . . . . . 8

4.1 Runtime for different data sets on two different computation environments 16

4.2 Time analysis for Different Data sets for different custer specification . . . 18

4.3 Time analysis for second Query . . . . . . . . . . . . . . . . . . . . . . 19

4.4 Outputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.5 Distribution overhead . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4.6 Raspberry Pi result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

vii

List of Figures

2.1 A vertex cut . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2 Edge Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.3 Neo4j Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3.1 Sub-Graph Formation. . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.2 Flow chart of cluster working . . . . . . . . . . . . . . . . . . . . . . . . 14

1

Chapter 1

Introduction

With the increase in information and data available for the digital computing devices it is

becoming more and more challenging to store, process and analyze this enormous amount

of data. Database is a generic name for the collection of information and its relations and

dependencies on other information. After relational database reaching to its highest limit,

computer researches started working on new data storage and computing paradigms that has

never been even thought of by humanity. Graph database is a way of storing information in

common data structure called Graph. In which we store information as nodes and edges of

the graph. This makes traversal over the data very easy and free. Dependencies are more

clear. But with increase in data sets it is becoming very difficult to perform computations

on these graph, it possible time complexities of these algorithms used is very high or the

execution time is too much. So researches worldwide are trying to find and develop new

paradigms in graph computations to lower the complexity or time by any means. Distribu-

tive and parallel computing played a great role for this purpose. Dividing computation over

number of connected machine and operating in parallel lowers the the total effective execu-

tion time.

The main focus of our project was to develop a cluster of GraphChi nodes [1]. GraphChi

is a disk based graph computation tool developed by research scholars: Aapo Kyrola at CMU

university. GraphChi performs various graph computation on a single machine in compara-

ble time against a large cluster of over thousand nodes. By breaking large graph into small

subgraphs and utilizing novel algorithms like parallel sliding windows method, I enables

2

GraphChi to execute a number of advanced data mining [2], graph mining [3], and machine

learning algorithms on very large graphs, using just a single consumer-level computer. It

process graph from the disk itself following this algorithm. Programs for GraphChi are

written in the vertex-centric model [4], proposed by GraphLab and Google Pregel. using

GraphChi changes and update written to edges are immediately visible to subsequent com-

putation, running vertex-centric programs asynchronously and in parallel [5]. It saves you

from the hassle and costs of working with a distributed cluster or cloud services because

promise of GraphChi is to bring web-scale graph computation, such as analysis of social

networks, available to anyone with a modern laptop. In some cases GraphChi can solve

bigger problems in reasonable time than many other available distributed frameworks. In

our project we developed a cluster of raspberry pis and configured GraphChi on each node

run multi-hop queries to evaluate the difference between single machine. As we have al-

ready seen that GraphChi on single node is effectively comparable than a cluster consisting

multiple machines. By running GraphChi in distributed manner in order to enhance the

performance of the novel algorithm. Various features, research outcomes, challenges and

working is discussed under different sections.

1.1 Objective

Objectives completed over the period of last one year i.e. August 2014 - June 2015

1. Configure Neo4j on a machine

2. Configure GraphChi on a node with similar hardware/software configuration

3. Run same multi-hop queries on both machines using different graph computation tools

4. Evaluate performance and time and space complexity comparison

5. configure cluster of Raspberry pis, configure GraphChi on each node

6. Write distributed multi-hop queries to perform same functionalities

7. Run and collect data about time and space complexities, Compare the performance

with existing data.

3

8. Conclude and summarize the performance of distributed GraphChi system.

1.2 Scope

As the information is increasing exponentially, it is happening for the first time in history

that we have enormous amount of data to store and process so humanity doesn’t have an

effective paradigm to handle this problem. This is a open problem and scholars worldwide

are viewing it from all angles. With GraphChi we can perform computations on large graph

data sets even comparable to huge clusters. If we can enhance its performance by distributing

it over a small cluster. We can reduce the hardware cost even for large scale computation by

building cost-effective GraphChi cluster using cheap computing devices such as Raspberry

pis. But scalability can still be a big issue.

4

Chapter 2

Background and Related Work

This chapter contains background and inspiration behind the project and related work done

in the field of graph computation and graph partitioning, two prime keywords for our project.

An asynchronous system for graph traversals on external and semi-external memory pro-

posed by Pearce et al [6]. Their solution does not allow changes to the graph but unlike

GraphChi, stores the graph structure on disk using the compressed sparse row format Vertex

values are stored in memory, and computation is scheduled using concurrent work queues.

Their system is designed for graph traversals, while GraphChi is designed for general large-

scale graph computation and has lower memory requirements. Another method is based

on simulating parallel PRAM algorithms, and requires a series of disk sorts, and would not

be efficient for the types of algorithms we consider. For this a collection of I/O efficient

fundamental graph algorithms in the external memory setting was proposed by Chiang et.

al. [7]. Many real-world graphs are sparse, and it is unclear which bound is better in prac-

tice. A similar approach was recently used by Blelloch et. al. for I/O efficient Set Covering

algorithms.

2.1 Graph partitioning

Graph partition is how to divide graph such that it is easy for an application or user to

use based on a specific scenario [8], is a huge problem and different approaches have been

discovered and taken into consideration for the project by out team. Some of the approaches

5

are discussed below.

2.1.1 PowerGraph

GraphLab developed a new way of partition the graph [9]. ”Think like a vertex”. A single

vertex is shared by different sub graphs to ensure the integrity of the edges and the relations,

the degree is split accordingly. Edges are given the high priority. Now that vertex can be part

of different partitions independently, Results can be joined after the execution on different

partitions is done. This approach is useful when we have restricted memory resources,

because we do not have to store extra information about the partitioning and the critical

edges that are shared between a cut. Vertex cut is shown in Figure 2.1.

Fig. 2.1: A vertex cut

2.1.2 Edge partitioning

A very basic approach is cutting graph from the edges! [10] shown in Figure 2.2, but in such

a way that we have minimum edge cuts to minimize the information stored about the cut.

Because edges having both end points in same sub graph can be processed in parallel but the

critical edges that are part of a cut we have to process in a sequential manner. That will take

more time and limit the parallelism. In case of million or billion of edges the information to

be stored and processed serially will be huge resulting into an inefficient system.

6

Fig. 2.2: Edge Partitioning

Fig. 2.3: Neo4j Graph

2.2 Neo4j

Neo4j is also a graph-database, written in Java. It a open source disk-based java persistence

engine which store information in generic data structure i.e. Graphs instead of using tables

methods to store data [11] [12]. Everything is stored in either node or edges. Nodes and

edges can have any number of attributes hich help Neo4j to traverse the graph and these

attributes also explain the relationship between different entities or information present in

the graph as shown in Figure 2.3. More to that Edges and nodes can be labelled which again

ease the graph traversal. To enable graph traversal and interact with node Cipher query

language is written which is solely inspired from basis structure of SQL [13]. Although

7

Neo4j os NoSQL database model.

2.3 GraphChi

The inspiration for this project is based on the research of two scholars at CMU: Aapo

Kyrola, who wrote a disk based graph computation system which operate over a single ma-

chine. This utility uses novel algorithms like parallel sliding window algorithms, graph

partitioning etc. to load graph into primary memory and perform computation. They com-

pared performance of GraphChi configured on a single mac mini machine with large cluster

consisting over fifty to one thousand machines, taking large graph data sets with number of

edges varying over few millions to billions. The output results of the comparison is shown

in table 2.1

ApplicationInput

graphGraph size Comparison

GraphChi

on Mac

Mini

Pagerank - 3 iterations twitter-2010 1.5B edgesSpark, 50 machines,

8.1 min13 min

Pagerank - 100 iterations uk-union 3.8B edges

Stanford GPS

(Pregel), 30 machines,

144 min

581 min

Web-graph Belief Propa-

gation (1 iter.)yahoo-web 6.7B edges

Pegasus, 100 ma-

chines, 22 min27 min

Matrix factorization

(ALS), 10 itersNetflix 99M edges

GraphLab, 8-core ma-

chine, 4.7 min9.8 min

Triangle counting twitter-2010 1.5B edgesHadoop, 1636 ma-

chines, 423 mins55 min

Table 2.1: Runtime for different data sets for different query apps

2.3.1 Features of GraphChi

1. Vertex-centric model

2. Asynchronous, parallel execution, with (optional) deterministic scheduling

3. Can run graphs with billions of edges, with linear scalability, on a standard consumer

grade machine

8

4. Works well on both hard-drive and SSD.

5. Evolving graphs, streaming graph updates

6. Easy to install, headers-only, no dependencies.

2.4 Working of Graphchi

GraphChi can perform computation on large graph data sets having over billion edges us-

ing parallel sliding window algorithm. It splits the graph in p number of parts termed as

shards [14].

2.4.1 Parallel Sliding Window

With only a small number of non-sequential disk accesses, PSW can process a graph with

mutable edge values efficiently from disk, while supporting the asynchronous model of com-

putation [15]. PSW processes graphs in three stages:

2.4.1.1 Loads the graph from disk

In PWS methos we split graph in disjoint intervals and associate shards with it. A shard

contains all the edges having destination in these interval, stored in the order of the source.

So when loading th graph it loads one of the p shards completely into the primary, termed as

memory shard. Then parts of the other shards are loaded into the memory that have in-edge

in the memory shard, termed as sliding shard. That make it a subgraph that is loaded into the

memory on which the computation is done. Similarly next shard is loaded into the memory

with respective sliding shards so the window that have in edges keep sliding for next edges.

2.4.1.2 Updates the vertices and edges

After fully loading interaval p subgraph into the memory from the disk, this algorithm ex-

ecutes the update-function for each vertex defines by user, in parallel. We enforce external

determinism to prevent adjacent vertices from accessing edges concurrently, As update-

functions can modify the edge values. It make sure that each execution of parallel sliding

9

window produces exactly the same result. This plays an important role for the accuracy of

the results. Critical edges are updated in sequential order. Non-critical vertices can be up-

dated safely in parallel. This solution, of course, limits the amount of effective parallelism.

For some algorithms.

2.4.1.3 Writes updated values to disk

When GraphChi is done updating and writing edge values, they need to be written back to the

disk. Because it needs to be visible to next execution interval. PSW can do this efficiently:

The edges are cached in memory ny loading them in large blocks from the disk. After creat-

ing subgraph for an interval, pointers to the cached blocks are used as reference to the edges.

That is, modifications to edge values directly modify the data blocks themselves. After fin-

ishing the updates for the execution interval, PSW writes the modified blocks back to disk,

replacing the old data. In this process the memory-shard is completely rewritten, while only

the active sliding window of each sliding shard is rewritten to disk. When PSW moves to

next interval, it reads the new values from disk, thus implementing the asynchronous model.

10

Chapter 3

Implementation and Working

Main focus of this project was large scale graph computation in a distributed way using

Graphchi, a single machine based graph computation framework. But before that perfor-

mance of Graphchi was evaluated with respective to existing graph computation tools. Later

in the next phase of the project to distribute computation over the cluster of Graphchi nodes,

main challenge was to divide the subjected graph, distribute the sub graphs over the cluster

and combining the results from different nodes in order ot get full output for a user query

accurately.Primary features and parts of the project are discussed below:

1. Performance evaluation and results.

2. Graph partition

3. Multi-hop query

4. Cluster of GraphChi Nodes

5. Raspberry Pi

3.1 Performance Evaluation

In the beginning performance of GraphChi was compared with that of Neo4j graph compu-

tation framework. This measurements were taken for single machines with same hardware

and software specifications. Respective nodes were configured with GraphChi and Neo4j.

11

Graph data sets of different sizes based on number of vertices and average degree of vertex

i.e. number of edges varying from 1 Million to few Billion edge were taken for the bench-

marking purposes. Graphs were taken in Edgelist format in GraphChi.

To query a graph with GraphChi framework, existing user programming apps based on

structure and libraries of GraphChi were used. These apps are written to be compatible with

working plan of the GraphChi it uses all the features like storing and excessing graph from

the disk, ensuring random excess to get efficient retrieval of data. Then as explained in work-

ing of Graphi sharding, loading into Ram, computations updating the nodes is done in order

to get the output of the user query performed using user programming app. On other hand

Neo4j takes graph input in .csv format. Graph in Neo4j contains more information about

nodes and relations between them. To query a graph with Neo4j, as it is a NoSQL database

so it has its own querying language called Cipher [13]. Same queries as GraphChi apps

were written in Cipher to perform on Neo4j graph. Time of preprocessing and Execution

were recorder for both for different size graph data sets and compare them after. With the

comparison it came out that for larger graphs GraphChi outrun the performance of Neo4j

with a considerable difference. Some of the numeric results are shown in table 4.1 in chapter.

3.2 Graph Partition

In order to develop a cluster of GraphChi nodes and distribute computation over this cluster,

a huge challenge was how to divide graph in an efficient manner to get accurate results with

least amount of computation and communication overheads. As discussed in chapter 2, dif-

ferent approaches were taken into consideration and implementation to test their respective

benefits and drawbacks for this project both theoretically as well as practically. Some of the

approaches were edge partitioning, vertex cut, Power Graph etc. Every partition paradigm

its different significances. But compatibility with existing GraphChi structure was a big is-

sue. So GraphChi also use partition algorithm to load parts of graphs into main memory

for computation i.e. Sharding was taken into consideration. As this algorithm is already

compatible with our framework and uses GraphChi libraries only.

12

3.2.1 Sub-Graph

As explained in Chapter 2, main inspiration behind this algorithm of partition is PSW. Firstly

Graph is divided into intervals based on the number of vertices, but in case of cluster it is

number of machines connected to server or Master node. Every interval then is connected to

a shard which contains all the in-edges, known as Memory Shard. But when it is loaded into

main memory some part of other shards i.e. Sliding shards are also loaded which contain

out-edges starting from the vertices in memory shard. we have all the information about the

memory shard nodes. It makes a complete graph independent of other parts of the parent

graph and can be used independently on one of the nodes in cluster. So memory shard

and sliding shards gives the sub graphs shown in Figure 3.1. Instead of loading them into

memory for computation, we store all sub graphs from the parent graph into graph files on

Master node to be distributed over the cluster for parallel computation.

Fig. 3.1: Sub-Graph Formation.

3.3 Multi-Hop query

For a multi-hop query, Friends of Friends was chosen as suggested by the initial objectives

of the project. A user programming app is written for this purpose taking advantage of

GraphChi libraries. This is a two step query. After getting Vertex-Id input from user it find

all the nodes having edges directed to the input vertex i.e. friends. Same query is again run

13

for all those output nodes from the first step. Final output gives search results with depth

equal to 2.

Fig. 3.2: Flow chart of cluster working

3.4 Cluster of GraphChi Nodes

To develop cluster nodes raspberry pis, small computing devices were used. Server-client

architecture was implemented for this. Cluster scripts were written in python using sockets

and ssh. Nodes were physically connected using Ethernet cables and switch. GraphChi was

configured on each node before performing any query. There is no communication between

clients. Master node process the graph and divide it into sub graphs using the sharding

algorithm explained above. Then it distribute sub graphs to the clients in the structure using

threads for parallel transfer of data. Now every client has an independent graph to work

on. Input is given as the id of vertex for which we want to find friends of friends. Server

send this input to client having that node in its interval and get back list of its friends by

performing search of single depth. Server then distribute this list on the network depending

on which client has what interval. Same query is run for those vertices on respective clients

in parallel fashion. Finally all the results from clients are collected and merged to get final

14

results using python scripts. working of the cluster is shown in Figure 3.2.

3.5 Raspberry Pi

Raspberry Pis are small and cheap computing devices. It comes with small computing

power. There are number of versions available for operating systems to be used in a rasp-

berry pi, Raspbian was used for this project. SD card was used for additional space required

by Graphchi to process and create graph files. In order to use them as a cluster static IP

address by changing its network files. These operating system comes with the support for

python, sockets etc., that were used in the cluster management scripts. After that GraphChi

was configured on each node, using them as client nodes.

15

Chapter 4

Experimental Studies and Results

In the first half of the project performance GraphChi was compared with Neo4j. Different

multi-hop queries were executed on different data sets varying the number of edges in order

to compute the total time of execution. The results are shown in Table 4.1.

Graph Algorithm Neo4j GraphChi

Twitter Social Network PageRank 684s 495s

Twitter Social Network Connected Components 643s 548s

Table 4.1: Runtime for different data sets on two different computation environments

It was very clear that GraphChi outperform Neo4j for large data sets having edges in

few millions to billions. For small graphs there was not any considerable difference. As

we are dealing with large data set of information hence it is safe to conclude that GraphChi

is better that Neo4j in single machine. Also referring to the original CMU research papers

GraphChi have a execution time comparable with cluster of machine having fifty to one

thousand nodes.

Then a cluster of Raspberry pis were set up. After configuring GraphChi on each node.

Queries like friends of friends were executed in a distributed manner. Cluster programs were

written in python to communicate and exchange data between server and client. Reason

behind using Server-client architecture is that we only need working client to communicate

with server which has all the information needed for the computation for example server

provide the vertex to find friends for and client can work independent of other nodes after

that. So By using this way the message passing and internode communication overhead was

16

neglected.

4.1 Algorithm Complexities

This section contains methamatical complexities of the various algorithms used for writing

programs and user apps.

Notations

NF→ Number of Friends of QV

N→ Number of shards or Intervals

TF→ Number of Friends of Friends Qv

M→ Number of Machines

Fi→ Number of Friends of Qv on machinei

4.1.1 For single Machine

For single machinw total time taken majorly depends on firstly degree of vertex queried plus

degree of the vertices returned as friends for input vertex. Execution Time, T = T1 +T2 +T3

where, T1 = running freinds query on Qv,

T2 = running freinds query on each friends of Qv,

T3 = Extracting the result of friends of friends

Let runtime of friends query be O(x)

T = O(X) + O(FX) + O(T F)

T = O(FX) + O(T F)

4.1.2 For Cluster

For cluster new factor of number of machines present also there. First find query vertex and

its friends that is in order of its degree. retreived results are then returned to the master.

after that same query is run for all vertex but they are distributed based on interval they are

present in. So Interval with maximum number of query vertices will become the deciding

17

factor and degree of vertex present in it. Execution Time, T = T1 + T2 + T3 + T4 + T5

where, T1 = finding machine containing Qv,

T2 = running freinds query on Qv,

T3 = creating sub-lists for different machines of the cluster

T4 = running friends query parallely on all machines,

T5 = extracting results from different machines.

Let runtime of friends query be O(X)

T = O(M) + O(X) + O(FM) + O(max(Fi) ∗ X) + O(T F)

T = O(FM) + O(max(Fi) ∗ X) + O(T F)

4.2 Cluster Performance

After developing distributed user programming app, computation were performed using dif-

ferent cluster specification with different number of nodes. Also queries were performed on

varying size data sets. Results of cluster with 1 to 4 machines (1 Server, 3 clients) are shown

in Table 4.2. It contains Preprocessing time as well as execution time to return the output.

Preprocessing is time taken by GraphChi to load graphs, sharding and create subgraphs, then

after sub graph distribution on each client set up its own GraphChi over received subgraph.

Nodes is the number of clients present in working cluster. These process was followed for

graphs of different sizes carrying number of vertices from approx. .37 Million to approx. 11

Million.

GraphSmall Medium Large

Vertex Id: 3 Vertex Id: 45784 Vertex Id: 45784

Nodes Preprocessing

Time(s)

Run

Time(s)

Preprocessing

Time(s)

Run

Time(s)

Preprocessing

Time(s)

Run

Time(s)

1 18.14642 54.3895 37.4096 551.1842 54.4182 523.206

2 90.9967 24.57306 113.8601 376.0392 145.1398 450.3001

3 95.9638 29.4926 127.3752 392.7315 160.0045 432.5887

Table 4.2: Time analysis for Different Data sets for different custer specification

18

4.2.1 Subsequent Query

Once the cluster is setup completely and for second query there will not be any preprocessing

overhead anymore. So another query was performed with other vertex id on previously setup

and processed system. Table 4.3 contains run times for second queries for same data sets.

To be noted there is no preprocessing time once the system is set up and done performing

one user query.

Graph Small Medium Large

Vertex Id 22190 43989 43989

Nodes Run Time(s) Run Time(s) Run Time(s)

1 154.5285 1282.0674 1208.2224

2 132.5277 1043.1365 815.8312

3 210.5277 858.1176 780.4372

Table 4.3: Time analysis for second Query

4.2.2 Query Output

This section contains output results for Vertices used for different graph data sets. Table 4.4

contains number of Friends and Friends returned by our FriendsofFriends user programming

app. Friends is th enumber of nodes returned by client as the output of first query then this

list will also contain vertex ids of al friends which will be distributed to other cllients to get

final number as well as vertex id of friends of friends.

Sr.No. vertex Id Friends Friends of

Friends

Graph Type

1 3 51 14022 Small

2 221290 123 651423 Small

3 45784 455 1644664 Medium

4 43989 1069 172509 Medium

5 45784 455 16801.7 Large

6 43989 1069 195400 Large

Table 4.4: Outputs

19

4.2.3 Distribution Time

On the initial setup of a graph, there is a one time distribution overhead that is transferring

sub-graphs to the clients nodes after partitioning. To minimize this time we implemented

threads so that subgraph file can be sent from server to all clients simultaneously. Table 4.5

contains distribution overhead in terms of time for different cluster specifications.

Number of

Nodes

Distribution

Time(s)

Graph

Type

Attributes

2 60.8105Small Vertices: 378, 049 Edges: 48, 723, 273

3 81.4019

2 82.7607Medium Vertices: 1, 051, 092 Edges: 67, 706, 979

3 154.0388

2 115.1917Large Vertices: 11, 316, 811 Edges: 85, 331, 846

3 Pending

Table 4.5: Distribution overhead

4.3 Raspberry Pi Cluster

Final expected output for this project was to develop a working prototype of GraphChi

cluster working on cheap small computing devices e.g. Raspberry pis. So we port developed

distributed Grapchi on configured raspberry pi cluster and performed friends of friends query

for a small graph and the data and output result are shown in Table 4.6.

GraphVertices: 45687

Edges: 18, 705, 331

Preprocessing Time 308.75346s

Run Time 417.54757

Distribution Time 127.67022s

Output Friends: 51

Vetex: 3 Friends of Friends:

13789

Table 4.6: Raspberry Pi result

20

Chapter 5

Conclusion and Future Work

Now we know that what kind of power GraphChi has for graph computation and how it

has enables us to perform large scale computations on a consumer level pc. Which is a

huge deal. Existing graph computational frameworks are either commmercial or have a

large cluster requirements. which makes GraphChi so special. It not only outrun a large

cluster performance but also cheap in terms of capital. In our project we tried to see if its

performance can be enhanced with the help of cluster and a distributive GraphChi functions.

Extracted results significantly show that query time can be further minimized by distributing

over a small cluster.

But with progress in this project many things came up, which can be taken into consid-

eration for further deleopment of this project some them are disscussed below:

1. This was the first attempt to divide graph but it leads to new way of partitioning graph

because we get total independent subgraph.

2. Network latency and physical media lead to more subgraph distribution overhead. If

we can decrease this latency by using on chip processing units it will effectively lower

the total execution time.

3. Our main focus was on social media graph analysis e.g. Friends of Friends query but

other apps can also be developed centered on machine learning and data mining or

e.g. Pagerank etc.

21

References

[1] A. Kyrola, “Introdunction to graphchi,” vol. 2, 2012, Description available at

https://github.com/GraphChi/graphchi-cpp.

[2] J. Frand, “Data mining: What is data mining?” 2011, Article available at

http://www.anderson.ucla.edu/faculty/jason.frand/teacher/technologies/palace/datamining.htm.

[3] Takashi Washio, and Hiroshi Motoda, “State of the art of graph-based data mining,”

Research Thesis, Osaka University, Japan.

[4] Yuanyuan Tian, and Andrey Balmin, and Severin Andreas Corsten, and Shirish

Tatikonda , and John McPherson, “From,” Master’s thesis.

[5] Robert Ryan McCune, Tim Weninger, and Greg Madey, “Thinking like a vertex: a

survey of vertex-centric frameworks for large-scale distributed graph processing,” Re-

search Thesis, University of Notre Dame, Jan 2015.

[6] Manuel Then, and Moritz Kaufmann, and Fernando Chirigati, and Thomas Neumann,

“The more the merrier:efficient multi-source graph traversal,” Research Thesis, New

York University, 2011.

[7] Michael T. Goodrich, and Lars Arge, and Nodari Sitchinava, “Parallel external memory

graph algorithms,” Research Thesis, University of California, 2012.

[8] Ulrich Elsner, “Graph partitioning - a survey,” Research Thesis, Technische Universitat

Chemnitz, Dec 1997.

[9] JosephE.Gonzalez, YuchengLow, HaijieGu, DannyBickson, and CarlosGuestrin,

“PowerGraph distributed graph - parallel computationon natural graphs,” Research

Thesis, CarnegieMellonUniversity, 2010.

[10] wiki, “Cut (graph theory),” 2012, http://en.wikipedia.org/wiki/Cut graph theory.

22

[11] Wikipedia, “Neo4j,” 2013, Description available at

https://github.com/GraphChi/graphchi-cpp/wiki/Introduction-To-GraphChi.

[12] lutovich, “Neo4j,” Code available at https://github.com/neo4j/neo4j.

[13] “Cipher query language,” Description available at

http://neo4j.com/docs/stable/cypher-query-lang.html.

[14] aapo Kyrola, “How graphchi works,” vol. 2, 2012, Description available at

http://en.wikipedia.org/wiki/Neo4j.

[15] Cagri Balkesen, and Nesime Tatbul, “Scalable data partitioning techniques for parallel

sliding window processing over data streams,” Research Thesis, ETH Zurich, Switzer-

land, 2011.

23

Curriculum Vitae

Name: Khushpreet Singh

Date of birth: 18 January, 1994

Education qualifications:

• [2011 - 2015] Bachelor of Technology (B.Tech),

Computer Science and Engineering,

Indian Indtitute of Technology,

Himachal Pradesh, India.

Permanent address:

H# 625, W.No 8, Labh Singh St.

Near bus stand, New court road

Mansa, Punjab, India, 151505.

Ph: +918988095924

24