Fast Graph Construction Using Auction Algorithm

Jun Wang and Yinglong XiaIBM T. J. Watson Research Center

1101 Kitchawan Rd, Yorktown Heights, NY 10598, USA{wangjun, yxia}@us.ibm.com

Abstract

In practical machine learning systems, graphbased data representation has been widelyused in different learning paradigms, rang-ing from unsupervised clustering to super-vised classification. Besides those applica-tions with natural graph or network struc-ture data, such as social network analysis andrelational learning, many other application-s often involve a critical step of the conver-sion from data vectors to a data graph. Inparticular, a sparse subgraph extracted fromthe original adjacency graph is required dueto both theoretic and practical needs. Previ-ous study clearly shows that the performanceof different learning algorithms, e.g., cluster-ing and classification, benefit from such s-parse subgraphs with balanced node connec-tivity. However, the existing graph construc-tion methods are either computationally ex-pensive or with unsatisfactory performance.In this paper, we introduce a scalable methodcalled auction algorithm and its parallel ex-tension to recover a sparse yet balanced sub-graph with significantly reduced computa-tional cost. Empirical study and compari-son with the state-of-art approaches clearlydemonstrate the superiority of the proposedmethod in both efficiency and accuracy.

1 INTRODUCTION

Graph based data representation treats data points asgraph nodes and builds pairwise edges between thesenodes which are often weighted by the similarity orcorrelations between the corresponding data samples.In the past decade, graph based data representationhas become very popular and widely used many ma-chine learning algorithms due to both practical needs

and theoretical advantage. First, under many practi-cal situations, the data come in a natural way of graphrepresentation, where the links or connections betweenitems can be treated as directed or undirected edges.Typical examples include social networks, the Inter-net, and citations networks [12, 21]. Second, for theconventional vector space based data representation, adata graph can be easily generated using pre-definedsimilarity or correlation measurement.

Given the popularity of data graph representation,many graph based machine learning algorithms havebeen developed, including graph based clustering andsemi-supervised learning algorithms. For instance,representative graph based clustering approaches in-clude those formulated as a graph cut problem [15, 20].In addition, graph has also been integrated into semi-supervised learning paradigm, resulting in a widerange of approaches, such as graph Laplacian regular-ization framework [2, 23, 25] and random walk basedmethods [1, 22]. Other popular methods include graphbased ranking algorithms [16, 24]. In many real ap-plications and systems, graph based approaches haveshown empirical success, especially for those with ag-nostic setting and given no prior knowledge of the datadistributions.

As indicated in [11, 14], constructing a suitable datagraph is the vital step for ensuring the learning perfor-mance. The sparse structure of the constructed graphis desired for learning algorithms to remain efficien-cy to memory usage and robustness to noisy samples.Given an adjacency graph, a typical way to recovera sparse graph without losing the global structure ofthe data is to build a neighborhood graph. The ob-jective is to connect each data point with its nearestneighbors and prune all other connections. Howev-er, since the nearest neighborhood method prune theedges through a greedy process, it can only producean asymmetric graph with some adhoc symmetriza-tion processes. As a result, the constructed neighbor-hood graph is often irregular and imbalanced, where

the learning performance could be significantly de-graded [11]. Therefore, this drawback of nearest neigh-bor method motivated one to consider matching tech-nique, such as b-matching, to build a balanced graphstructure [10, 11]. Instead of achieving symmetriza-tion by post-processing, b-matching sets the symmetryas additional constraints and directly extracts a sym-metric solution. But this modification imposes muchhigh computation and memory cost and the one of theknown fast approximated solution using belief propa-gation has the space complexity of O(n) and the timecomplexity O(n2.5) [9], where n is the number of graphnodes (refer to Table 1).

Considering the trade off between computational effi-ciency and performance, in this paper, we propose afast and scalable solution for constructing nearly bal-anced graphs using the auction algorithm proposedin [3]. In order to remain performance superiorityand reduce computation cost, our method tends toextract an approximately balanced graph though p-reserving the regularity as much as possible. We showthat this method has the space complexity O(n) andtime complexity O(|E| ·max |aij |/ϵ), where max |aij |/ϵis a constant determined by the data and the learn-ing parameter ϵ. It is much less expensive than theexisting regular graph construction methods, e.g. b-matching technique. In addition, the auction algo-rithm based graph construction is naturally paralleliz-able and therein the computational efficiency can befurther improved using multiple processes. We pro-vide extensive experiments and comparison study withthe state-of-the-arts on both synthetic and real bench-mark datasets. The experimental results clearly showthat the graphs created by parallel auction algorithm(PAA) provide superior performance than neighbor-hood graphs for different machine learning algorithm-s. Although PAA graphs have slightly lower qualitythan b-matching graphs, we observe that PAA methodreduces the computational cost by hundreds of timesthan b-matching method. Therefore, the PAA methodis very suitable for handling large scale applicationswith massive data graphs.

The remainder of this paper is organized as follows. InSection 2, we give a brief introduction of related workon graph construction process. A new graph construc-tion and sparsification method using a parallel auctionalgorithm is presented in Section 3. The experimentsare reported in Section 4 followed by conclusions anddiscussions in Section 5.

2 RELATED WORKS

Although graph based machine learning methods havebeen extensively studied and explored, there have been

limited efforts for building effective data graphs inthe community. Although some recent methods usesampling techniques to approximate large scale graph-s [13], it still remains as a challenging issue to effi-ciently construct large scale graphs with good qual-ities, such as sparsity and regularity. In this section,we will provide a brief introduction about related workfor graph construction and then motivate our proposedmethod.

Given a data set X, graph construction aims in con-verting the data to a weighted graph G = {X,E,W}where graph nodes are the samples and W ∈ Rn×n isa weight matrix of edges E. As summarized in [11],two important steps for graph constructions are spar-sifying the adjacency matrix A = {aij} ∈ Rn×n andassigning weights to edges. In many practical settings,the adjacency matrix is computed using a similarityfunction for all pairs of nodes, i.e. aij = sim(xi,xj).For other applications, such as social network analysisand relational learning, the adjacency or connectionmatrix A might be given in a natural way without theabove process of computing pair-wise similarity. In thesecond step, graph sparsification tries to recover a s-parse binary matrix B = {bij} ∈ Bn×n to preserve theglobal structure of data using local information. Thefinal edge matrix can be easily derived as W = B •A,where the symbol • indicates element-wise multipli-cation. Such an extract sparse subgraph often leadsbetter learning performance and requires less storageand computation cost [10, 11].

The most popular method for constructing a sparsegraph is the nearest neighbor (NN) approach, includ-ing different variants such as k-nearest neighbor andϵ-nearest neighbor methods. Since ϵ-nearest neigh-bor often generate fragile subgraphs and does notprovide satisfactory performance, kNN graph remain-s more popular in many applications [14]. Briefly s-peaking, kNN graph construction performs a greedysearch process to select the edges with the most signif-icant weights for each node, and remove all the otherinsignificant ones. The objective of kNN approach in-deed solves the following maximization problem

maxbij

∑i

∑j

bijaij (1)

s.t.∑j

bij = k, (2)

bii = 0, bij ∈ {0, 1}, ∀i, j = 1, · · · , n. (3)

Since the greedy search for k nearest neighbors doesnot guarantee to produce symmetric B, it usuallyneeds some postprocessing to achieve symmetrizationby max(B,B⊤) or min(B,B⊤). However, this adhocsymmetrization often leads to imbalanced connectivityfor nodes, which could seriously degrade the learning

Table 1: The time and space complexity for different graph construction approaches. Assume the input adjacencygraph has n nodes and |E| edges. The compared methods are: Loopy Belief Propagation (LBP) [8], SufficientSelection Belief Propagation (SSBP) [9], Nearest Neighbor Method, and the Auction Algorithm.

Method Time Space

LBP Method O(n|E|) O(n+ |E|)SSBP Method O(n2.5) O(n)

Nearest Neighbor Method O(n2) O(1)

Auction Method O(|E| ·max |aij |/ϵ) O(n)

performance, as observed in [11].

As a valuable alternative for extracting sparse graphstructure, weighted b-matching technique [17] has beenapplied to build graph with balanced node connectiv-ity. Similarly to the objective of kNN approach, b-matching recovers a sparse graph with the maximizededge weights. Furthermore, b-matching enforces thesymmetry of connectivity through directly imposing aset of constraints bij = bji. Essentially, b-matchingis a generalized version of the well-known linear as-signment problem, which can be solved by Hungari-an method with a cubic complexity O(n3) (n = |X|)in time. Various methods have been proposed tosolve b-matching problem with less memory and timecost, including the reduction based method [7] andbelief propagation based approaches [8, 9], for whichthe computational complexity is summarized in Ta-ble 1. Although b-matching method yields clear ad-vantage for different machine learning algorithms, in-cluding spectral clustering and semi-supervised clas-sification [10, 11], it is a much more expensive pro-cedure than the intuitive kNN method. Due to bothtime and memory constraints, the existing b-matchingbased graph construction is infeasible for large scaledatasets.

3 AUCTION ALGORITHM FORGRAPH CONSTRUCTION

In this section, we describe the procedure to utilizean auction algorithm to identify bn significant edgesfrom the given bipartite graph corresponding to theadjacency matrix A. In particular, each node is con-nected with at most b edges. If given an arbitrarygraph, it is straightforward to convert it to a bipartitegraph though creating a shadow node for each graphnode and only connect the original graph nodes to theshadow nodes. In order to utilize distributed comput-ing systems to process massive data graphs, we alsodescribe the parallelization technique for the proposedgraph construction method.

3.1 Auction Algorithm for LinearAssignment

Here, we first describe the solution for a single-edgematching problem using an auction algorithm, andthen provide general extensions. For a standard max-imum 1-matching problem (also known as a linear as-signment problem) over a bipartite graph with a non-negative weight matrixA, the objective can be writtenas:

maxB

tr(A⊤B) (4)

s.t. B1T = 1, BT 1 = 1, B ∈ Bn×n,

where 1 = (1, 1, · · · , 1)T . This linear program can beinterpreted as selecting a permutation matrix B, sothat after permutation of A, the sum of entries on thediagonal is maximized. The edges corresponding tothe permutation matrix B is as known as a matchingin A. Thus, the solution of this program can be usedto sparsify A with constraints of regularity. To gen-eralize the standard linear assignment problem to theobjective of extracting a balanced graph where each n-ode connects to b edges, we can simply replace 1 withb = b · 1. However, this solution for such a b-matchingproblem can be approximated by solving the abovelinear assignment problem for b/2 times and removingthe selected edges after each iteration.

As discussed earlier, we seek fast and scalable solutionsfor graph sparsification due to the demand of process-ing large scale graphs in many domains. Conventionalapproach for solving Eq. 4 is based on increasing aug-menting paths, which leads to low concurrency andlimited scalability [3, 19]. Thus, we propose to usethe auction algorithm introduced by Bertsekas in [3].Since the constrained problem in Eq. 4 can be con-verted into a dual form, the auction algorithm dealswith the above linear programming by solving its dualproblem. In particular, we are solving the followingdual problem of the original primal linear program:

minp,π

1T p+ 1Tπ (5)

s.t. 1pT + π1T ≥ A,

where p = (p1, p2, · · · , pn)T and π = (π1, π2, · · · , πn)T

are known as the dual vectors. The constraints1pT + π1T ≥ A indicate the element wise relation-ship between the matrices 1pT + π1T and A, i.e.(1pT + π1T )ij ≥ aij , i, j = 1, · · · , n. The auction algo-rithm computes the dual variables through an iterativeauction process. Intuitively, we view each row i as abuyer and each column j as an object; aij is the ben-efit of object j to buyer i; pj is the price of object j(initialized to 0 at the beginning); πi is the profit forbuyer i, defined as (aij − pj), in case object j is as-signed to i. The auction algorithm proceeds as follows:Each buyer i bids for an adjacent object j that offer-s the highest profit, i.e., maxj∈adj(i)(aij − pj), whereadj(i) denotes the set of adjacent nodes of i. The bidis the difference between the highest profit and thesecond highest profit. An object j is assigned to thebuyer offering the highest bid, say buyer i. Thus, rowi and column j are matched and price pj is increasedby the bid. Price increment makes an object expensivefor competitors, so that they can choose other objects.The above steps are repeated for all unmatched rowsuntil every row is matched or the matching does notchange.

The naive auction algorithm described above has adefect caused by price war [3]. When the highestand second highest profits are equal for some buyers,the bid becomes 0 and the object price remains un-changed. Therefore, buyers can indefinitely competefor the same object without increasing its price. Theobject will be alternately assigned to them and theauction will not progress. To overcome this issue, asmall positive scalar ϵ is added to the price of an ob-ject once it is assigned to a buyer. A smaller ϵ leads tobetter accuracy; a greater ϵ results in an acceleratedauction process.

3.2 Auction for Graph Sparsification

The auction algorithm described in Section 3.1 selectsa single edge for each node while maximizing the totalweight of the selected edges. For selecting b edges,there are two straightforward approaches to generalizethe auction. The first approach is to perform auctionmultiple times. Every time the algorithm selects anedge for each node, if exists, without having two edgesconnect to the same node. The other approach is toselect up to b edges in each iteration process, followingthe same criteria for edge selection. The details arediscussed in below.

Note that the above algorithm does not ensure that ifa node i connects to a shadow node j′, node j will alsoconnect to node i′. This can cause adverse impact ongraph regularity. We propose a heuristic strategy toreduce such negative impact. When such inconsistence

Algorithm 1 Fast Graph Construction Using ParallelAuction Algorithm

Graph partitioning: partition graph G into Ldisjoint subgraphs {Gl}Ll=1 with weight matrices{Al}Ll=1, where Al = {alij}Initialization: Dual vectors p = 0, initial π = 0,the set of selected edges M = ∅while price keeps changing do

[1]. For each subgraph corresponding to theweight matrix Al = {alij}, compute for nodes inXl that have not identified b edges:

jl1 = argmax(alij − plj), ∀j ∈ adj(i)

......

jlb+1 = argmax(alij − plj), ∀j ∈ adj(i), j = j1, · · · jb

[2]. Remove earlier selected edges that conflict

M = M\{(i′, jlt) ∈ M, ∀1 ≤ t ≤ b, i′ ∈ Xl}

[3]. Add current selected edges

M = M∪ {(i, jlt) ∈ M, ∀1 ≤ t ≤ b

[4]. Update price

pljlt= alijlt

− alijlt+1+ pljlt+1

+ ϵ,∀t = 1, · · · , b

[5]. Synchronize price globally plj = maxPt=1 ptj

end while

occurs, saying node j selects node i′ instead of node i′,we compare the percentile of edge (j, i′) with respectto all incident edges of j and the percentile of (i, j′)with respect to all incident edges of i. By percentile,we mean the index of the selected edge among all in-cident edges sorted in descending order. We acceptthe edge with smaller percentile while reject the otherone. The intuition that we select edges according tothe percentile instead of edge weight is that all inci-dent edges for the boundary nodes in a graph can beweighed much higher than even the lightest incidentedge of an internal node.

3.3 Parallelization

In this subsection, we provide the parallelization s-trategy of the auction algorithm for graph sparsi-fication. Given the weight matrix A of a bipar-tite graph, we first partition A into L disjoint sub-matrices, where each sub-matrix Al = {alij} is of size

nl×n,∑L

l=1 nl = n. Therefore, sub-matrix Al is theweight matrix of a bipartite subgraph Gl, which con-tains disjoint node sets Xl and Yl. Although A can

Table 2: Average running time (seconds) on the synthetic bipartite and unipartite graphs using different methodsfor constructing sparse graphs. The graphs have nodes from 100 to 2000 and the value of b is set as 10 uniformly.The compared methods include Loopy Belief Propagation (LBP) [8], kNN, and the proposed parallel auctionalgorithm (PAA).

n Bipartite Unipartite

|E| LBP kNN PAA |E| LBP kNN PAA

100 5× 103 0.2 8× 10−4 4× 10−3 9.9× 103 0.3 4× 10−3 1.1× 10−2

200 2× 104 2.8 3× 10−3 1.5× 10−2 3.98× 104 0.7 0.01 0.04

500 1.25× 105 18.7 0.02 0.11 2.495× 105 6.2 0.07 0.28

1000 5× 105 37.9 0.11 0.53 9.99× 105 23.2 0.39 1.26

2000 2× 106 609.7 0.47 2.42 3.998× 106 111.4 1.31 5.39

also be partitioned by columns or into checkerboardstructures [18], the above horizontal partitioning alongrow direction is more natural since large scale sparsegraphs/matrices are often stored in the compress s-parse rows format. After partitioning, we can rewritethe linear assignment problem in Eq. 4 as:

maxL∑

l=1

∑i∈Xl

∑j∈Yl

aijbij (6)

s.t.∑j∈Yl

bij = 1, ∀i ∈ Xl, l = 1, · · · , L

L∑l=1

∑i∈Xl

bij = 1, ∀j ∈ Yl, bij ≥ 0, ∀i, j.

Eq. 6 implies a straightforward parallelization of auc-tion algorithm for graph sparsification. In Algorithmchart 1, we summarize the major steps of parallel auc-tion for graph sparsification. Here we set the initialvalues for p and π are both 0. The maximum numberof edges allowed for a node is denoted by b and theset for selected edges is M, which is initialized as anempty set ∅.

The above algorithm allows concurrent auction on amaximum of L processors, each of which handles onesingle bipartite subgraph. The only communicationbetween processors occurs when synchronizing pricevector in step 5 of the above parallel algorithm. Notethat the solution of the other dual variable π is updat-ed implicitly, which is given by πl

i = alij − plj , where(i, j) is selected. The parallel auction typically acceler-ates computation due to the concurrent auction activ-ities in each sub-matrix. However, it does not changethe complexity. That is, in the worst case (the inputgraph is a chain), it is equivalent to a serial auctionprocess.

3.4 Complexity Analysis

The time complexity for auction algorithm depends onthe number of auction iterations and the workload for

each iteration. In an auction process, a price of nodeincreases at least for ϵ [3], i.e., the minimum bid in-crement. Since the profit for bidder i that matchedwith object j is given by (aij −pj), the profit becomesnegative after |aij |/ϵ iterations, i.e., i will seek oth-er objects for possibly higher profit. In each itera-tion, we visit at most |E| edges (the number typicallydecreases dramatically after a few iterations), so theoverall complexity is O(|E|max |aij |/ϵ). During theauction process, the only information we must keep isthe price of objects. There are n objects, so the spacecomplexity is O(n). Note that the space for storingthe data or graph itself is not counted for evaluatingspace complexity. In Table 1, we summarize the timeand space complexity of the proposed parallel auctionalgorithm based graph construction process and com-pare with other methods. From this analysis, it is easyto see the proposed parallel auction algorithm (PAA)based method is extremely efficient for large scale s-parse graph data, which is indeed very common in re-al applications, such as social network and geometryanalysis [6].

4 EXPERIMENTS

In this section, we provide empirical study of the pro-posed parallel auction algorithm (PAA) based graphconstruction method and compare with the-state-of-art approaches. In particular, we are interested incomparing with the belief propagation (BP) basedmethods [8, 9] since these method tends to generatehigh quality graphs with balanced node connectivity,similar as our proposed method. Note that there aretwo variants of the BP approach, loopy belief propaga-tion (LBP) and a speed-up version, sufficient selectionbelief propagation (SSBP). However, in the empiricalstudy, we notice that these two methods share verysimilar computational performance. Hence, we onlyreport the results of LBP for comparison. In addi-tion, we also implement the nearest neighbor searchbased graph construction using the standard sorting

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

(a)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

(b)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

(c)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

(d)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

(e)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

(f)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

(g)

Figure 1: Illustration of the constructed graphs and clustering results over the toy datasets using differentmethods: a) original data; b) k-nearest neighbor graph (k = 8); c) b-matching graph (b = 8); d) PAA graph.Different color indicate the extract clusters using: e) kNN graph; f) b-matching graph; g) PAA graph.

process as another baseline. Extensive experimenter-s over both synthetic and real benchmark datasetsare performed to evaluate the computational efficiencyand performance for different learning algorithms, asdiscussed below.

4.1 Speed Comparison

We first evaluate the running time of different graphconstruction methods. Similar to the settings in [8],here we randomly generate bipartite and unipartitegraphs with 100 ≤ n ≤ 2000 nodes, where the weightsof graph edges are independently sampled from a u-niform distribution U(0, 1). For bipartite graph, eachof the two disjoint node sets has n/2 nodes. For con-structing a sparse graph, the degree of each nodes ofthe extracted sparse graph is uniformly set for all theexperiments. For instance, in kNN approach, the valueof k is set as 10, and accordingly b = 10 for b-matchingmethods. For fair comparison, all the experiments areconducted on the same platform with identical hard-ware and software environment.

For the LBP method, we use the default parameters inthe software package 1, except setting the maximumiteration as 5000. Table 2 shows the comparison of therunning time for the synthetic data. It is obvious thatkNN provides the most efficient graph constructionmethod due to its fast greedy search process. How-

1http://www.cs.umd.edu/bert/code/bmatching/

ever, comparing LBP and PAA, both of which tendto generate balanced graphs, PAA improves the speedof graph construction in the magnitude of 102. In ad-dition, we found that the propose method has morestable running performance for different random data,while both LBP method relies more on the propertiesof the synthetic graph data.

4.2 Performance Comparison

Graph representation has been widely used for differ-ent machine learning algorithms, including clusteringand classification. In this subsection, we present theperformance comparison study of clustering and clas-sification through using different graph constructionapproaches. For clustering analysis, spectral cluster-ing has been developed and shown effective in prac-tice [15]. In a standard spectral clustering process,there are three key steps. The first step is to con-struct a data graph that connects sample points. Thenone has to perform eigen-decomposition over the graphLaplacian. Finally, K-means algorithm is applied toderive the final clusters. In this study, we use the toydata [11] to construct different graphs, namely kNNgraph, b-matching graph, and the graph built by theparallel auction algorithm (PAA-graph). Figure 1(a)-1(d) show the original data and different graphs. Thenwe feed these graph into standard spectral clusteringalgorithm to compare the results.

6 8 12 16 20 240

0.02

0.04

0.06

The Degree of Graph Nodes

Err

or

Ra

te

KNN graphb−Matching graphPAA graph

Figure 2: The performance comparison of spectralclustering using different graphs. The horizontal ax-is indicates the number of connected edges for graphnodes and the vertical axis shows the error rate of clus-tering results.

Figure 2 shows the error rate for the clustering result-s. The number of connected edges for graph nodesvaries from 6 to 24. For all the tests, we use the sameGaussian kernel based weighting scheme with the k-ernel bandwidth informally set as 0.1. Clearly, kNNgraph provides the worst performance and b-matchinggraph has the best performance. The proposed parallelauction based graph construction method has slight-ly worse performance than b-matching. However, itsignificantly increases the by the magnitude of 102 inpractice. Figure 1(e)-1(g) shows the examples of con-structed sparse graphs using different approaches. Dueto the greedy search process, kNN method generateshighly irregular graph and breaks the data structureby creating more cross-manifold edges. In contrast, b-matching and the proposed PAA method create morebalanced graphs and have much less edges across thetwo clusters. When the node degree increases, it iseasy to generate more cross-cluster edges. Hence, allthe clustering performance of all three graph construc-tion methods accordingly decreases.

In addition, we also compare the performance of graphbased semi-supervised learning for classification tasks,i.e. classifying USPS handwritten digits [4] and facerecognition [5]. In particular, we apply the methoddeveloped by Zhou et.al in [23] as the testbed for d-ifferent graph construction methods. In the tests, wefix the number of node degree as 7 and use the sameweighting scheme, similar to the settings used in theabove clustering experiments. The number of initiallygiven labels is varied from 4 to 20 for digits data and 3to 18 for the face data. For each setting, we randomlyrun 50 independent trials and compute the average er-ror rates of the classification results. Figure 3(a) and3(b) show the performance curves with error bars fordigits classification and face recognition, respectively.

4 8 12 16 20 240.02

0.04

0.08

0.12

The Number of Labels

Err

or

Rate

kNN graph

b−Matching graph

PAA graph

(a)

3 6 9 12 15 180

0.08

0.16

0.24

The Number of Labels

Erro

r R

ate

kNN graph

b−Matching graph

PA graph

(b)

Figure 3: The performance comparison of graph basedsemi-supervised learning method using different graphconstruction methods: a) digits classification; b) facerecognition. The horizontal axis indicates the numberof initially given labels and the vertical axis shows theerror rate of classification results.

Again, b-matching approach builds the most balancedgraph and therein achieves the best performance. Theproposed PAA method has slightly worse performancebut almost as fast as kNN method, which tends tocreate the highly imbalanced graphs.

4.3 Evaluation of Parallel Efficiency

Since all the above experiments involve the graphswith relatively small scales, there is no extra bene-fit to parallelize the algorithm. However, when han-dling large scale data, it is very important to performparallel process for fast graph construction. Note thatthe time complexity of the PAA based graph construc-tion is linear with respect to the number of edges.This property makes the approach very suitable tothe applications with large scale sparse data graph-s. In order to validate the parallelization efficiency ofthe proposed method for such applications, here weused two real benchmark datasets about circuit simu-lation problems from the sparse matrix collection [6].The first graph data “ASIC 680ks” contains 682, 712

(a) (b)

Figure 4: The visualization of the two large graph da-ta: a) ASIC 680ks, and b) ASIC 680k.

nodes and 2, 329, 176 edges (after reverse process) 2,and the second data “ASIC 680k” has 682, 862 nodesand 3, 871, 773 edges (after reverse process) 3. Figure 4presents the visualization of these two large graphs4.From this figure, it is easy to see that both graph-s are heavily irregular and the boundary nodes arewith extremely sparse connectivity. Here we apply theproposed PAA method to recover sparse and balancedgraphs from the original graph data.

Our experiments are performed using a computer clus-ter on which each compute node contains two 2.33 GHzIntel Xeon E5410 quadcore processors. The eight coresshare 32 GB DDR2 RAM running at 800 MHz. TheOS is Ubuntu 9.10 with GCC 4.10 installed. All codeswere compiled with -O3 level option. We spawn par-allel processes using MPICH2 to evaluate the parallelperformance of the proposed PAA based graph con-struction method. In addition, we utilize MPICH2 sothat our proposed technique can scale up to multiplecompute nodes when the scale of the input graph islarge enough. The graphs are partitioned evenly andlaunched to each process with equal number of nodesalong with all the neighbor information.

For both graph data, graph construction is performedfor two scenarios: b = 2 and b = 4. For those nodeswith less than b edges in the original graph, all theconnected edges will be preserved. We vary the num-ber of used processes from 1 to 8, where each processis hosted by a separate core. The computational costincludes the time for generating parallel process, parti-tioning the graph, assigning tasks to each process, andperforming the auction process (The time for loadingdata is not considered). Table 3 provides the runningtime in seconds for each tested case. From this table, itis easy to see that the proposed PAA-based graph con-

2http://www.cise.ufl.edu/research/sparse/matrices/Sandia/ASIC 680ks.html

3http://www.cise.ufl.edu/research/sparse/matrices/Sandia/ASIC 680k.html

4The visualization examples are available from the web-site: http://www.cise.ufl.edu/research/sparse/matrices/

Table 3: Evaluation of the parallel efficiency of theproposed PAA-based graph construction method. Thenumbers give the computational cost in seconds usingdifferent number of processes.

Data ASIC 680ks ASIC 680k

# of processes b = 2 b = 4 b = 2 b = 4

1 3.09 18.92 1.76 11.62

2 2.74 18.73 1.49 10.83

4 2.70 11.60 1.33 7.29

8 2.47 8.92 1.26 5.03

struction method achieves lower execution time whenmultiple processes were used. However, the speedupis sublinear instead of linear because the extra timeused for parallelization and communication. In gener-al, when performing the graph construction in 8 par-allel processes, the time cost will be reduced from onequarter to more than half. Especially for more com-plex problems (e.g., b = 4), the speed up is usuallymore significant.

5 CONCLUSION

Data graph has been widely used for different machinelearning and data mining algorithms. It is critical tobuild high quality graphs with affordable cost. Com-monly, one aims in extracting sparse yet regular graphfrom data for both theoretic and practical advantages.Popular methods for constructing sparse graphs in-clude nearest neighbor method and b-matching algo-rithm. The former is very efficient since it essentiallyperform greedy search to pick up the significant graphedges. However, it often generates highly imbalancedgraphs with unsatisfactory performance. The later canprovide fairly balanced regular graphs but suffers fromextremely high computational cost.

In order to remain computational efficiency, whileachieving certain properties of the extracted graph,such as sparsity and regularity, in this paper, we pro-posed a novel graph construction method through ex-ploring auction algorithm. In particular, the proposedmethod is almost as fast as kNN method, while main-taining satisfactory regularity property. Experimentalresults show that the parallel auction algorithm basedgraph construction provide comparable performanceto b-matching algorithm, but improves the speed inthe magnitude of 102. Finally, we also provide theempirical study of the parallelization efficiency overlarge data sets with up to 680K nodes. One of ourfuture direction is to extend the proposed graph con-struction to semi-supervised and supervised scenarios,where partial label information are given to help im-prove accuracy of the auction process.

References

[1] A. Azran. The rendezvous algorithm: Multi-class semi-supervised learning with markov ran-dom walks. In Z. Ghahramani, editor, Proceed-ings of the International Conference on Machinelearning, pages 49–56, Corvalis, Oregon, USA,June 2007.

[2] M. Belkin, P. Niyogi, and V. Sindhwani. Mani-fold Regularization: A Geometric Framework forLearning from Labeled and Unlabeled Examples.Journal of Machine Learning Research, 7:2399–2434, 2006.

[3] D. Bertsekas and D. Castanon. A forward/reverseauction algorithm for asymmetric assignmentproblems. Computational Optimization and Ap-plications, 1:277–297, 1993.

[4] M. Breitenbach and G. Grudic. Clusteringthrough ranking on manifolds. In Proceedingsof the 22nd international conference on Machinelearning, pages 73–80, 2005.

[5] M. Breitenbach and G. Grudic. Clusteringthrough ranking on manifolds. In Proceedingsof the 22nd international conference on Machinelearning, pages 73–80, 2005.

[6] T. A. Davis and Y. Hu. The university of flori-da sparse matrix collection. ACM Trans. Math.Softw., 38(1):1:1–1:25, 2011.

[7] H. Gabow. An efficient reduction technique fordegree-constrained subgraph and bidirected net-work flow problems. In Proceedings of the fifteenthannual ACM symposium on Theory of computing,pages 448–456, 1983.

[8] B. Huang and T. Jebara. Loopy belief propaga-tion for bipartite maximum weight b-matching.In Int. Workshop on Artificial Intelligence and S-tatistics, 2007.

[9] B. Huang and T. Jebara. Fast b-matching viasufficient selection belief propagation. In Proceed-ings of the Fourteenth International Conferenceon Artificial Intelligence and Statistics, 2011.

[10] T. Jebara and V. Shchogolev. B-matching forspectral clustering. In Proceedings of the Eu-ropean Conference on Machine Learning, pages679–686, 2006.

[11] T. Jebara, J. Wang, and S.-F. Chang. Graphconstruction and b-matching for semi-supervisedlearning. In Proceedings of the 26th Annual Inter-national Conference on Machine Learning, pages

441–448, Montreal, Quebec, Canada, June 2009.ACM.

[12] D. Lazer, A. Pentland, L. Adamic, S. Aral, A.-L.Barabasi, D. Brewer, N. Christakis, N. Contrac-tor, J. Fowler, M. Gutmann, T. Jebara, G. King,M. Macy, D. Roy, and M. Van Alstyne. Computa-tional social science. Science, 323(5915):721–723,6 February 2009.

[13] W. Liu, J. He, and S. Chang. Large graph con-struction for scalable semi-supervised learning. InProceedings of the 27th International Conferenceon Machine Learning, pages 679–686, 2010.

[14] M. Maier, U. von Luxburg, and M. Hein. Influ-ence of graph construction on graph-based clus-tering measures. In D. Koller, D. Schuurmans,Y. Bengio, and L. Bottou, editors, Advances inneural information processing systems, volume 22,pages 1025–1032. MIT Press, 2009.

[15] A. Ng, M. Jordan, and Y. Weiss. On spectralclustering: Analysis and an algorithm. Advancesin neural information processing systems, 2:849–856, 2002.

[16] L. Page, S. Brin, R. Motwani, and T. Winograd.The pagerank citation ranking: Bringing order tothe web, 1999.

[17] C. Papadimitriou and K. Steiglitz. Combinatorialoptimization: algorithms and complexity. DoverPubns, 1998.

[18] C. Park, W. An, K. Pattipati, and D. Kleinman.Distributed auction algorithms for the assignmen-t problem with partial information. In 15th In-ternational Command and Control Research andTechnology Symposium, pages 1–15, 2010.

[19] E. J. Riedy and J. Demmel. Sparse datastructures for weighted bipartite matching. InSIAM Parallel Processing for Scientific Comput-ing, pages 1–2, 2004.

[20] J. Shi and J. Malik. Normalized cuts and imagesegmentation. Pattern Analysis and Machine In-telligence, IEEE Transactions on, 22(8):888–905,2000.

[21] X. Shi, J. Leskovec, and D. McFarland. Citing forhigh impact. In Proceedings of the 10th annualjoint conference on Digital libraries, pages 49–58,2010.

[22] M. Szummer and T. Jaakkola. Partially labeledclassification with markov random walks. In Ad-vances in Neural Information Processing Systems17, volume 14, pages 945–952. MIT Press, Cam-bridge, MA, 2002.

[23] D. Zhou, O. Bousquet, T. N. Lal, J. Weston, andB. Scholkopf. Learning with local and global con-sistency. In S. Thrun, L. Saul, and B. Scholkopf,editors, Advances in Neural Information Process-ing Systems, volume 16, pages 321–328. MITPress, Cambridge, MA, 2004.

[24] D. Zhou, J. Weston, A. Gretton, O. Bousquet,and B. Scholkopf. Ranking on data manifolds. Ad-vances in neural information processing systems,16:169–176, 2004.

[25] X. Zhu, Z. Ghahramani, and J. Lafferty. Semi-supervised learning using gaussian fields and har-monic functions. In Proceedings of InternationalConference on Machine Learning, pages 912–919,2003.