Methodology for analysis of the research trend: a case ... · PDF fileMethodology for analysis of the research trend: a case ... computer science. Most traditional research ... analysis

Methodology for analysis of the research trend: a case study of security researchin South Korea

Hyoungshick Kima, Ji Won Yoonb

aComputing Laboratory, University of Cambridge, Cambridge, [email protected]

bStatistics department, Trinity College Dublin, Dublin 2, [email protected]

Abstract

We propose a new research evaluation method to identify relatively important journals/proceedings and underlyingpatterns from open databases for a specific field such as computer science. Most traditional research evaluationmethods underestimate inherent characteristics and criteria of each field despite their importance. The key insightis to analyse the publication pattern of true experts in the related field and then use it as a guideline for evaluatingscientific production. We demonstrate the practicality of our method by analysing the scientific production by Koreanresearchers in a security subject from 2006 to 2008. Through this empirical analysis, we can also observe the dynamiccharacteristics of the security research trend in Korea.

Key words: Research Evaluation, Research Trend, Korean Security Research, Publication Analysis, NetworkAnalysis

1. Introduction

1.1. MotivationFor the last decade, the South Korean government have invested a lot of budgets to make Korean research universi-

ties and institutes to be globally competitive and to produce more high-quality research results in Korea. In particular,a special research funding program, Brain Korea 21 (BK 21) project, designed to strengthen the competitiveness ofuniversities in Korea, launched by the Korea Ministry of Education [39]. It provides fellowship funding to graduatestudents and professors in research groups at top Korean universities. As a result, the quantitative growth of researchpublications from Korean research universities and laboratories has been shown perceptibly. For example, the numberof the research publications from Seoul National University (SNU), was over 4,000 in 2005, thus the quantity wasnot less than that of world-class universities such as Stanford, Johns Hopkins, and UC Berkeley. In 2005, articlepublications by SNU faculty approached about 50% of the publications from Harvard, the university with the highestnumber of total Science Citation Index (SCI) publications worldwide in 2005. Considering that SNU!s productionof published articles was about 18% of Harvard!s in 1995, the growth of article publications can be considered anoutstanding accomplishment [39]. However, such a quantitative growth at the macro level do not necessarily meanthe improvement of research performance in a specific field. For example, prior literatures insist on that traditional re-search evaluation methods based on a database such as ISI bibliographic database1 is not appropriate for some studiessuch as computer science [41].

In general the scientific literature is organized in relatively discrete clusters of journals/proceedings. However,most traditional research evaluation methods underestimate each field!s own characteristics and criteria. For example,in computer science, prestigious conferences hosted by ACM/IEEE societies are more preferred to present originaland important results by the researchers. However, these publications are not included in SCI, one of indexes in ISI

1The ISI citation databases, which include the Arts and Humanities Citation Index (A& HCI), Science Citation Index (SCI) and Social SciencesCitation Index (SSCI), have for decades been used as de facto standard tools for conducting publication and citation analyses [23].Preprint submitted to Journal of Informetrics July 21, 2009

database [24]. In addition, many citation analyses do not consider the relative weights although articles in a field suchas biochemistry can be frequently cited compared to those in other fields such as engineering. From these examples,we directly observe two main problems as follows:

• How can we select a set of prestige journals/proceedings for a specific subject?

• How can we rank the relative importance or positions of journals/proceedings within themselves?

Our goal is to extract underlying patterns from journals/proceedings data to evaluate scientific production in aspecific subject well. Ideally, scientific production can be e!ectively evaluated by experts in the same field. The keyinsight is to analyse true experts! publication pattern and then use it as a guideline for evaluating scientific productionof target research group.

1.2. ContributionAlthough research evaluation techniques on publication data have been well developed, the research and models in

a specific subject such as computer science are still in their infancy. This study provides a new direction for researchevaluation by analysing publication pattern. In particular, we make the following two contributions:

• We propose a new method to evaluate research quality by comparing the target publication pattern with an idealpublication pattern. For comparison of publication patterns, we construct their relationship graphs in Section 2and then analyse the similarity of the graphs.2 As one of reasonable metrics, we suggest how many nodes arecommonly included into both relationship graphs at the same time in Section 3.3. In addition, we suggest howto make a pseudo-ideal publication pattern by collecting the known top (global) researchers! publication data inSection 3.1 since there is no such an ideal publication pattern in practice.

• We empirically analyse the trend of the publications for security field in South Korea and evaluate it using theproposed method in Section 3. The e!ects were measured by comparing the relationship graphs constructed byKorean researchers! publication data with those defined by the world!s top researchers! publication data duringthe 2006 – 2008 period.

2. Evaluation method

We evaluate the target researchers! publication results by comparing them with the desirable results. Since it isdi"cult to compare these datasets directly, we transform them into the graphs to maintain the publication patternswell.

Given a set of target researchers T in a specific subject, we have basically the following steps to construct theuseful graphs:

1. Select a set of pseudo-ideal researchers P in the same field of T randomly or systematically.2. Collect the scientific production by T and P during the same period, respectively.3. Generate two bipartite graphs GT and GP with these collected publication data, whose nodes are divided into a

set of authors A and a set of journals/proceedings J and an edge (a, j) means that the author a published a paperin the journal (or proceeding) j for a " A and j " J (See Figure 1-(a)).

4. Construct two J-projected graphs GJT and GJ

P compressed by J-projection. The J-projection means a networkcontaining only nodes in J, where two nodes are connected when they have at least one common author (SeeFigure 1-(b)).

5. Evaluate GJT by comparing it with GJ

P.

2We note that these graphs are di!erent from the co-authorship (or collaboration) networks [27] of researchers where two researchers are con-sidered connected if they have co-authored one or more papers together. Unlike the co-authorship networks, we have interested in the relationshipbetween journals/proceedings.

2

In the step 1, if we select the researchers randomly, we can use sampling schemes. In addition, the normalizationis required if |T| ! |P|. For simplicity, we assume |T| = |P| in this paper. In the step 2, the research results canbe practically obtainned from the on-line databases such as Digital Bibliography & Library Project (DBLP) [18] forcomputer science and MEDLINE [22] and Scopus [38] for biomedical research. In the step 4, J-projection is well-known technique so-called one-mode projecting to show the relations among a particular set of nodes [14, 45]. Figure1 shows an example of one-mode projecting. In this paper, we use the simplest projecting method to project thebipartite network onto an un-weighted graph without taking into account of the repeated frequency of the relationshipbetween journals/proceedings.3

(a) A bipartite graph (b) A J-projected graph

Figure 1: an example of one-mode projecting: a set of nodes (pink) are removed and the bipartite graph is transformed.

3. Empirical analysis

3.1. Experimental dataAs a case study, we analyse security research in South Korea from 2006 to 2008. We use a sample set since

it is practically infeasible to collect all publications related to security. To obtain a reasonable sample set, we havebasically the following steps:

1. Select top conferences related to security field where the conferences are held in South Korea.2. Randomly select n researchers from the program committee members of the selected conferences in South

Korea.

In selecting conferences, the prior knowledge is required.4 We select two conferences, “International Workshop onInformation Security Applications” (WISA) [32] and “International Conference on Information Security and Cryptol-ogy” (ICISC) [31], considering their relatively large scales and long history compared to other conferences. Also, wedefine the sample size as 20 (n = 20). We assume that 20 active researchers are enough to show the characteristics ortrend per year. Let T be a set of randomly selected researchers.

In the similar manner, we obtain a reasonable sample set of global researchers by using the international topconferences for security, “IEEE Symposium on Security and Privacy”, “ACM Conference on Computer and Commu-nications Security” and “Usenix Security Symposium” instead of WISA and ICISC. These conferences are selectedunder the conference ranking of well-known web sites [37, 35, 36]. Let P be a set of the randomly selected researchersfrom these conferences.

We collect T!s and P!s publication results from 2006 to 2008, respectively. For simplicity, we only consider thebibliographic information of the security realm indexed by DBLP since the DBLP server provides most bibliographicinformation on major computer science journals and proceedings.

3We describe the problem of un-weighted graph in Section 4.4We discuss it in Section 4.

3

3.2. Bipartite graphsWith the collected publication data, we construct the bipartite graphs per year of each research group from 2006

to 2008. We can see the resulting graphs in Figure 2. In this figure, pink and green nodes represent authors and theirpublications. We use acronyms of journals/proceedings as green nodes! identifiers (See Appendix A.). Also we userandom numbers as the identifiers of pink nodes for the authors’ anonymity. From these bipartite graphs of Figure

(a) Korean researchers (2006) (b) Global researchers (2006)

(c) Korean researchers (2007) (d) Global researchers (2007)

(e) Korean researchers (2008) (f) Global researchers (2008)

Figure 2: Bipartite graphs from 2006 to 2008: pink (upward) and green (downward) nodes represent authors and their publications

2, we can extract meaningful characteristics and we summarize them in Table 1. Since some researchers in T and Pdo not have any publications in the DBLP database during 2006 – 2008 period, we can only draw between 14 and 17authors who have at least one publication in the related year among 20 sampled researchers. We compared the numberof conferences and journals published by Korean and global researchers. The ratio of journal

!!journals

!journals/proceedings

"of the

Korean researchers has been increased while the number of types of journals/proceedings has been decreased fromyear to year.

3.3. J-projected graphsBy J-projection in Section 2, we construct J-projected graphs from the bipartite graphs in Section 3.2. We can

see J-projected graphs in Figure ??. In addition, we merge “Korean researchers!” and “Global researchers!” graphs4

|A| |J| ! journals/proceedings ! journals Ratio of journals2006 (Korean) 14 40 77 12 0.1562007 (Korean) 16 43 61 20 0.3282008 (Korean) 14 37 51 23 0.4512006 (Global) 14 58 73 24 0.3292007 (Global) 16 46 59 9 0.1532008 (Global) 17 55 68 13 0.191

Table 1: Summary of publication data: |A| and |J| represent the number (cardinality) of the authors and their publications, respectively. We have|A| < 20 since we plotted only authors who has at least one publication.

(a) 2006 Korean (b) 2007 Korean (c) 2008 Korean

(d) 2006 Global (e) 2007 Global (f) 2008 Global

Figure 3: J-projected graphs of Fig. 2

to represent common nodes visually in Figure 4. In this figure, the red and blue edges set represent the J-projectedgraphs by Korean researchers and by global researchers, respectively. Figure 4 represents the network topologiesconsisting of journals/proceedings that Korean and global researchers have mainly published from 2006 to 2008. Inthis Figure, we can see two groups share only a small number of common nodes visually.

This fraction of common nodes between two group can be used a simple research evaluation metric. We analysehow many nodes are commonly included into both the target J-projected graph with the pseudo-ideal J-projectedgraph. We define the metric as follows:

Definition Ratio of common nodes– Given two J-projected graphs GJT = (VT,ET) and GJ

P = (VP,EP), we then

5

(a) 2006 (b) 2007 (c) 2008

Figure 4: J-projected graphs: The red and blue edges set represent the J-projected graphs by Korean researchers and by global researchers,respectively

Ratio Common nodes2006 0.032 ACNS, ISPEC, FC2007 0.083 ACIS, ACNS, FAW, ICISS, JNCA(J), Pairing2008 0.072 CCS, ESORICS, JUCS(J), TISSEC(J), WCNC

Table 2: Common nodes in the projected graphs

define the Ratio of common nodes as

Ratio(GJT,G

JP) =

VT#

VP

VT$

VP.

We can see the experimental results of this metric in Table 2. We observe that all ratios of the common nodes arestill under 10%.

3.3.1. Graph propertiesWe examine several structural properties of J-projected graphs (See Table 3). Network diameter is the maximum

distance between nodes in the network [33]. Network density is a normalized version of the average number of neigh-bours which indicates the average connectivity of a node in the network. Network density describes the overall level ofinteraction between all journals/proceedings. Network centralization reflects the extent to which interactions are con-centrated in a small number of journals/proceedings rather than distributed equally among all journals/proceedings.

! nodes ! edges average diameter density centralization clusteringdistance coe"cient

2006 (Korean) 39 180 2.000 5 0.243 0.493 0.7052007 (Korean) 40 128 2.103 4 0.164 0.610 0.7402008 (Korean) 30 99 1.993 3 0.228 0.495 0.8402006 (Global) 58 181 3.019 6 0.109 0.305 0.6122007 (Global) 38 123 2.077 4 0.175 0.700 0.9012008 (Global) 44 155 2.297 4 0.164 0.437 0.760

Table 3: J-projected graphs! properties

6

Networks whose topologies resemble a star have a centralization close to 1, whereas decentralized networks are char-acterized by having a centralization close to 0 [7]. The clustering coe"cient C, which was introduced by Watts andStrogatz, is one of parameters used to characterise the topology of networks [42]. Clustering coe"cient gives theprobability that two nearest neighbours of the same node are also mutual neighbours. In undirected networks, theclustering coe"cient of the node i with ki edges is given by the ratio between the number of edges Ei that actually ex-ist between the adjacent ki nodes and the total number ki(ki # 1)/2 of such edges that could exist in the neighbourhoodof i as follows:

Ci =2Ei

ki(ki # 1).

The clustering coe"cient of the whole network is the average of all individual Cis. Two nodes having a commonneighbour are also likely to be neighbours when the average clustering coe"cient C is large.

From Table 3, we observe that the network size of Korean researchers! J-projected graphs are approximatelydecreasing from year to year.

3.3.2. Central nodesBy ranking the journals/proceedings according to their positions in the network, we can interpret it as the relative

prominence of journals/proceedings embedded in the graphs. Several measures capture variations on the notion ofa node!s importance in a graph. We compute the conventional centrality metric values such as degree, eccentricity,closeness and betweenness in each graph, and then identify nodes with high metric values. Without loss of generality,in the case of a disconnected graph, we only consider the largest connected component in the graph since somecentrality metrics are meaningful for connected networks only.

Firstly, we compute high degree nodes in each J-projected graph. In general, high connectivity in the network isthe most important factor to determine a “influential” member in the network. We can see the experimental results inFig. 6 in Appendix B.

Secondly, we compute high eccentricity nodes in each J-projected graph. The eccentricity of a node v, e(v), iscomputed as follows [33]:

e(v) = maxu"V

1distance(v, u)

.

If the eccentricity of the node u is high, this means that all other nodes are in proximity. In contrast, if theeccentricity is low, this means that there is at least one node that is far form the node u. We can see the experimentalresults in Fig. 7 in Appendix B.

Thirdly, we compute high betweenness nodes in each J-projected graph. Let "st denote the number of the shortestpaths from s " V to t " V where "ss = 1. Let "st(v) denote the number of shortest paths from s " V to t " V passingthrough v " V . The betweenness of a node v, b(v), is computed as follows [13, 4]:

b(v) =%

s!v"V

%

t!v"V

"st(v)"st.

The betweenness centrality of a node reflects the amount of control that this node exerts over the interactions ofother nodes in the network [44]. This measure favours nodes that join communities (dense sub-networks), rather thannodes that lie inside a community. We can see the experimental results in Fig. 8 in Appendix B.

Fourthly, we compute high closeness nodes in each J-projected graph. The closeness of a node v, c(v), is computedas follows [29]:

c(v) =1

&u"V distance(v, u)

.

The closeness centrality is a measure of how fast information spreads from a given node to other reachable nodesin the network. We can see the experimental results in Fig. 9 in Appendix B.

In this paper, we define a central node as a node whose all observed metric values (degree, eccentricity, closenessand betweenness) are greater than the average values of each metric. For example, “IEICET(J)” is a central node, but“EUC” is not a central node in J-projected graph by Korean researchers in 2006 since the betweenness value of the

7




Figure 5: Comparison of nodes with high centrality

8

node “EUC” is less than the average betweenness in the network (See Appendix B.). Fig. 5 shows the central nodes ineach J-projected graph. We observe that there is no common central node between “Korean researchers!” and “Globalresearchers!” graphs.

3.4. DiscussionWe analysed publication patterns of Korean researchers in a security subject from 2006 to 2008. Interestingly,

although the number and the ratio of publication to the journal has been increased, the number of journal types isdecreasing in Korea as years go. That is, it is shown that the variation of the topics is reducing in Korea as time flows.

We also observed that the number and the ratio of publication to the journal in Korea has been dramaticallyimproved from 2007 in Table 1. We make a conjecture on the reason of this situation influenced from the fact that mostconference proceedings including those published in Springer!s Lecture Notes in Computer Science series (LNCS)have not been included in SCI from 2007. Korean researchers have seek the alternatives of such proceedings in thejournals from 2007 since scientific production in South Korea is generally evaluated by counting SCI publications.

We compared the journals/proceedings that Korean researchers have mainly focused on with those that top (global)researchers have focused on by projecting the bipartite graphs into projected graphs 3.3. In Fig. 4, we find thatKorean research group and global research group share only a small fraction of journals/proceedings. That is, Koreanresearchers and global researchers are publishing their papers in di!erent set of journals or conferences even thoughthey are working in the same subject. However, the number of shared journals/proceedings are increasing as yearflows so two groups shares more journals/proceedings in 2008 than in 2006.

Finally, in the comparison of central nodes, we observe that there is no common journals/proceedings with highcentrality between Korean research group and global research group. Under the strong assumption that a globalresearch group is close to ideal research group, we claim that Korean research group will have to exert itself morethan it currently does to publish many papers in journals/proceedings with high centrality (e.g. CCS) in the globalresearchers! graphs as shown in Fig. 5 (b), (d) and (f).

4. Limitation

Our work is primarily intended to demonstrate how to analyse the di!erence of the publication patterns and trendsbetween two di!erent research groups. We have not explicitly considered on the research quality since the results ofour metrics of this paper may not give so enough evidence to measure the quality between two groups (although wecan guess). The proposed analysis of publication pattern can be, however, a useful supplement to evaluate researchquality rather than as a replacement for traditional research evaluation methods based on publication and citationquantities.

The most challenging issue to achieve this goal is how to make an ideal publication pattern which can be comparedwith publication pattern of the target group. In practice, it is infeasible to explore such an ideal publication pattern.Instead, we practically suggested to construct a pseudo-ideal publication pattern by collecting publication data byknown internationally top (global) researchers in the same field. Still, this selection criteria has deficiency in that itis highly influenced by subjective factors such as preference for research topics and reputation. Thus, we need todesign unbiased sampling scheme by carefully considering several factors such as sampling method, sample size, andnormalization. In this paper, we simplified these important issues.

In addition conference (or journal) publication is strongly related to geographical and political factors in realworld. In this paper, we do not consider these factors.

We simply project the bipartite network onto an un-weighted graph. In the projected graph, it is generally in-evitable to lose a certain information. Un-weighted graphs can not maintain the information about the repeatedfrequency of relationship. For example, the identical projected graph is constructed when an author or two authorspublished papers in X and Y journals. However, in practice, this weight information may be important to evaluate theresearch results e!ectively.

We also consider a rather simplistic assessment such as counting common nodes between target and pseudo-idealgraphs. However, we need to extend this metric in analysing the similarity or the di!erence between publicationpatterns as important lines for future work.

9

5. Related work

The use of statistical indicators including publication frequency in research evaluation were emerged in the 1960sand 1970s [19], and is widely in use today due to easy access to the relevant databases. The publication and citationanalyses using on-line database provide robust and useful output measures of activity and performance of scientificresearch and have become standard tools for research evaluation. Therefore, individual institutions and researchers arenow classified on the basis of scientific productivity, usually established using the number of papers, the impact factorsfor the journals and the number of citations [1]. However, some methodological problems of research evaluation atthe macro level remain unresolved [34, 25].

One of the main issues is based on the fact that it is di"cult to evaluate the research performance of individualscientists at the micro level. To overcome this limitation, Hirsch proposed a new citation index called h-index forrating the research output of individual scientists [16] and many scientists believed that it is an objective measure ofindividual research achievement that is transparent, unbiased and easily measurable [43, 9]. The h-index representsthe highest number of papers a scientist has published that have each received at least that number of citations:for example, a researcher with an h of 50 has written 50 papers that have each had at least 50 citations. The h-index depends on both the number of a scientist!s publications, and their evaluation by the corresponding scientificcommunity. Many variations and extensions of this model have been proposed [3, 5, 8].

There were many case studies to evaluate scientific production in a specific subject. We mainly reviewed relatedto the research evaluation in computer science [21, 24, 41]. In particular, Meyer et al. [24] introduced several criteriaand principles to evaluate research quality in computer science. They warned that publication counts, weighted ornot, must not be used as indicators of research value. Since traditional research evaluation based on a bibliographydatabase such as ISI is not proper in computer science, most previous studies focused on the analysis of social networksof researchers such as co-citation networks [6, 11, 2, 40] and co-authorship networks [26, 15, 17, 10].

Co-authorship networks are an important class of social networks and have been used extensively to determine thestructure of scientific collaborations and the status of individual researchers. Many co-authorship networks have beenstudied to investigate the structure of scientific collaborations [27, 28, 12, 30, 20]. Our work is an extension of networkbased publication analysis by focusing on the relationship and topological positions between journals/proceedingsunlike those between researchers in co-authorship networks.

6. Conclusion

In this paper, we propose a new method to evaluate research performance of a specific group by comparing thetarget publication pattern with an ideal publication pattern. We can construct relationship graphs from the publicationdata that we mainly interested in and then analyses several properties of the constructed relationship graphs. Our goalis to analyse not the publication and citation quantities but the publication pattern and the research trend.

As a case study, we empirically analysed the publication patterns of Korean researchers in a security subjectfrom 2006 to 2008. We can learn many important characteristics such as the relative importance of some jour-nals/conferences. In addition we can approximately measure the di!erence between Korean researchers and top(global) researchers in the security field under the strong assumptions in Section 4.

There exist a lot of potentials in this novel approach. First of all, this approach can provide the dynamics ofthe research trend in a certain subject. We can see what is the major trend each year since every journal/conferencehave their own characteristics in the subject. This implies that weak and strong points of the research subject areextracted from the results and we can find where we need to invest more to make balance in research. In addition, asan extension, we will develop a measure to calculate the gap (di!erence) of the subject between two selected groups.

References

[1] J.A.S. Almeida, A.A.C.C. Pais, and S.J. Formosinho. Science indicators and science patterns in europe. Journal of Informetrics, 3(2):134 –142, 2009.

[2] Y. An, J. Janssen, and E. E. Milios. Characterizing and mining the citation graph of the computer science literature. Knowl. Inf. Syst.,6(6):664–678, 2004.

[3] M. G. Banks. An extension of the hirsch index: Indexing scientific topics and compounds. Scientometrics, 69:161, 2006.[4] U. Brandes. A faster algorithm for betweenness centrality. Journal of Mathematical Sociology, 25:163–177, 2001.

10

[5] T. Braun, W. Glanzel, and A. Schubert. A hirsch-type index for journals. Scientometrics, pages 169–173, 2006.[6] C. Chen and L. Carr. Trailblazing the literature of hypertext: Author co-citation analysis (1989–1998). In HYPERTEXT ’99: Proceedings of

the tenth ACM Conference on Hypertext and hypermedia : returning to our diverse roots, pages 51–60, New York, NY, USA, 1999. ACM.[7] J. Dong and S. Horvath. Understanding network concepts in modules. BMC Systems Biology, 1(1), 2007.[8] L. Egghe. Theory and practise of the g-index. Scientometrics, 69(1):131–152, April 2006.[9] L. Egghe. Mathematical study of h-index sequences. Information Processing & Management, 45(2):288–297, 2009.

[10] E. Elmacioglu and D. Lee. On six degrees of separation in dblp-db and more. SIGMOD Rec., 34(2):33–40, 2005.[11] C. Erten, P. J. Harding, S. G. Kobourov, K. Wampler, and G. Yee. Exploring the computing literature using temporal graph visualization. In

in Conference on Visualization and Data Analysis (VDA, pages 45–56, 2003.[12] I. Farkas, I. Derenyi, H. Jeong, Z. Neda, Z. N. Oltvai, E. Ravasz, A. Schubert, Barabasi, and T. Vicsek. Networks in life: Scaling properties

and eigenvalue spectra. Physica A: Statistical Mechanics and its Applications, 314(1-4):25–34, November 2002.[13] L. C. Freeman. A set of measures of centrality based on betweenness. Sociometry, 40(1):35–41, March 1977.[14] J. Guillaume. Bipartite graphs as models of complex networks. Lecture Notes in Computer Science, 3405:127–139, 2005.[15] A. E. Hassan and R. C. Holt. The small world of software reverse engineering. In Reverse Engineering, 2004. Proceedings. 11th Working

Conference on, pages 278–283, Nov. 2004.[16] J. E. Hirsch. An index to quantify an individual’s scientific research output. PNAS, 102(46):16569–16572, November 2005.[17] D. B. Horn, T. A. Finholt, J. P. Birnholtz, D. Motwani, and S. Jayaraman. Six degrees of jonathan grudin: a social network analysis of the

evolution and impact of cscw research. In CSCW ’04: Proceedings of the 2004 ACM conference on Computer supported cooperative work,pages 582–591, New York, NY, USA, 2004. ACM.

[18] M. Ley. The dblp computer science bibliography: Evolution, research issues, perspectives. In SPIRE 2002: Proceedings of the 9th Interna-tional Symposium on String Processing and Information Retrieval, pages 1–10, London, UK, 2002. Springer-Verlag.

[19] L. Leydesdor!. The evaluation of research and the evolution of science indicators. current science, 89(9), 2005, 1510-1517. 2005.[20] X. Liu, J. Bollen, M. L. Nelson, and H. Van deSompel. Co-authorship networks in the digital library research community. Information

Processing & Management, 41(6):1462–1480, 2005.[21] P. Lukowicz, E. A. Heinz, L. Prechelt, and W. F. Tichy. Experimental evaluation in computer science: A quantitative study. Journal of

Systems and Software, 28:9–18, 1994.[22] MEDLINE. http://medline.cos.com/, 2009.[23] L. I. Meho. The rise and rise of citation analysis, Dec 2006.[24] B. Meyer, C. Choppy, J. Staunstrup, and J. Leeuwen. Research evaluation for computer science. Communications of the ACM, 52(4):31–34,

2009.[25] Henk F. Moed. Citation Analysis in Research Evaluation. Information Science and Knowledge Management. Springer, December 2005.[26] M. A. Nascimento, J. Sander, and J. Pound. Analysis of sigmod’s co-authorship graph. SIGMOD Rec., 32(3):8–10, 2003.[27] M. E. Newman. Scientific collaboration networks. ii. shortest paths, weighted networks, and centrality. Physical Review E, 64(1):016132,

Jun 2001.[28] M. E. Newman. The structure of scientific collaboration networks. Proc Natl Acad Sci U S A, 98(2):404–409, January 2001.[29] M. E. Newman. A measure of betweenness centrality based on random walks, 2003.[30] M. E. Newman. Coauthorship networks and patterns of scientific collaboration, 2004.[31] International Conference on Information Security and Cryptology. http://www.icisc.org/, 2009.[32] International Workshop on Information Security Applications. http://www.wisa.or.kr/, 2009.[33] H. Per and H. Frank. Eccentricity and centrality in networks. Social Networks, 17(1):57 – 63, 1995.[34] T. Phelan. A compendium of issues for citation analysis. Scientometrics, 45(1):117–136, May 1999.[35] Computer Science Conference Ranking. http://www.cs-conference-ranking.org/conferencerankings/topicsi.html, 2009.[36] Computer Security Conference Ranking and Statistic. http://faculty.cs.tamu.edu/guofei, 2009.[37] Security Conference Ranking. http://www.doc.ic.ac.uk/ cd04/ranking.html, 2007.[38] Scopus. http://info.scopus.com/, 2009.[39] J. Shin. Building world-class research university: The brain korea 21 project. Higher Education, 2009.[40] A. Sidiropoulos and Y. Manolopoulos. A new perspective to automatically rank scientific conferences using digital libraries. Information

Processing & Management, 41(2):289 – 312, 2005.[41] J. Wainer, E. C. Xavier, and F. Bezerra. Scientific production in computer science: A comparative study of brazil and other countries.

Scientometrics, 2009.[42] D. J. Watts and S. H. Strogatz. Collective dynamics of ’small-world’ networks. Nature, 393(6684):440–442, June 1998.[43] G. Wolfgang. On the h-index - a mathematical approach to a new measure of publication activity and citation impact. Scientometrics,

67(2):315–321, 2006.[44] J. Yoon, A. Blumer, and K. Lee. An algorithm for modularity analysis of directed and weighted biological networks based on edge-

betweenness centrality. Bioinformatics, 22(24):3106–3108, 2006.[45] T. Zhou, J. Ren, M. Medo, and Y. Zhang. How to project a bipartite network? Physical Review E, 2007.

11

A. List of journals/proceedings

Acronym Journals/Conferences full nameACIS ACIS International Conference on Computer and Information ScienceACISP Australasian Conference on Information Security and PrivacyACNS Applied Cryptography and Network SecurityACSAC Annual Computer Security Applications ConferenceAFRICACRYPT AFRICACRYPTAHSWN(J) Ad Hoc & Sensor Wireless NetworksAINA Advanced Information Networking and ApplicationsAlgo(J) AlgorithmicaAMC(J) Applied Mathematics and ComputationAMSTA Agent and Multi-Agent Systems Technologies and ApplicationsAPNOMS Asia-Pacific Network Operations and Management SymposiumAPWeb Asia-Pacific Web ConferenceARES Availability, Reliability and SecurityASIACRYPT ASIACRYPTASIAN Asian Computing Science ConferenceASPLOS International Conference on Architectural Support

for Programming Languages and Operating SystemsATC Autonomic and Trusted ComputingAWIC Atlantic Web Intelligence ConferenceCANS Cryptology and Network SecurityCCS Conference on Computer and Communications SecurityCGO Symposium on Code Generation and OptimizationCGVR International Conference on Computer Graphics and Virtual RealityCHI Computer Human InteractionChinaF Science in China Series F Information SciencesCIS Computational Intelligence and SecurityCISIS International Conference on Complex, Intelligent and Software

Intensive SystemsCIT Conference on Computer and Information TechnologyCMS Communications and Multimedia SecurityCN(J) Computer NetworksColCom Collaborative ComputingComCom(J) Computer CommunicationsCompSec(J) Computers & SecurityCOMSWARE COMmunication System softWAre and MiddlewaRECoNEXT Conference on Emerging Network Experiment and TechnologyCPM Combinatorial Pattern MatchingCRYPTO International Cryptology ConferenceCSCWD Computer Supported Cooperative Work in DesignCSF IEEE Computer Security Foundations Symposium /WorkshopCSI(J) Computer Standards & InterfacesCT-RSA The Cryptographer!s Track at RSA Conference (CT-RSA)DASC Symposium on Dependable Autonomic and Secure ComputingDASFAA Database Systems for Advanced ApplicationsDBSec Database SecurityDF IFIP WG11.9 Publications - Digital ForensicsDIMVA Detection of Intrusions and Malware & Vulnerability Assessment

12

Acronym Journals/Conferences full nameDKE(J) Data & Knowledge EngineeringDRM ACM Digital Rights Management WorkshopDSN Dependable Systems and NetworksDSP(J) Digital Signal ProcessingDSS(J) Decision Support SystemsEDBT International Conference on Extending Database TechnologyEDOC Enterprise Distributed Object Computing ConferenceEMC Ershov Memorial ConferenceENTCS(J) Electronic Notes in Theoretical Computer ScienceESCAPE Combinatorics, Algorithms, Probabilistic and

Experimental MethodologiesESORICS European Symposium on Research in Computer SecurityEUC Embedded and Ubiquitous ComputingEuroPKI European Public Key Infrastructure WorkshopEuroVis IEEE TCVG Symposium on VisualizationEVOTE E-Voting and IdentityFAW Frontiers in AlgorithmicsFBIT Frontiers in the Convergence of Bioscience and

Information TechnologiesFC Financial CryptographyFDTC Fault Diagnosis and Tolerance in CryptographyFOCS Annual Symposium on Foundations of Computer ScienceFSE Fast Software EncryptionGLOBECOM IEEE GLOBECOMHCI Human-Computer InteractionHPCC High Performance Computing and CommunicationsICALP International Colloquium on Automata, Languages and ProgrammingICC IEEE International Conference on CommunicationsICCS International Conference on Computational ScienceICCSA Computational Science and Its ApplicationsICDCN International Conference of Distributed Computing and NetworkingICDCS International Conference on Distributed Computing SystemsICDE International Conference on Data EngineeringICDM IEEE International Conference on Data MiningICDM IEEE International Conference on Data MiningICHIT International Conference on Hybrid Information TechnologyICICS International Conference on Information and Communication SecurityICIP International Conference on Image ProcessingICISC Information Security and CryptologyICISS International Conference on Information Systems SecurityICIW International Conference on Internet and Web Applications

and ServicesICML International Conference on Machine LearningICOIN International Conference on Information NetworkingICPADS International Conference on Parallel and Distributed SystemsICUCT Ubiquitous Convergence TechnologyICWS International/European Conference on Web ServicesICYCS International Conference for Young Computer ScientistsIEICET(J) IEICE TransactionsIH Information Hiding Workshop

13

Acronym Journals/Conferences full nameIJES(J) International Journal of Embedded SystemsIJIS(J) International Journal of Information SecurityIJSN(J) International Journal of Security and NetworksIJSNET(J) International Journal of Sensor NetworksIPCCC IEEE International Performance, Computing, and

Communications ConferenceIPDPS International Parallel (and Distributed) Processing SymposiumIPL(J) Information Processing LettersIPSN Information Processing in Sensor NetworksIRI Information Reuse and IntegrationISC Information Security Conference /WorkshopISCA International Symposium on Computer ArchitectureISCC International Symposium on Computers and CommunicationsISCI(J) Information SciencesISCIS International Symposium on Computer and Information SciencesISE IFIP WG11.8 PublicationsISI Intelligence and Security InformaticsISPA International Symposium on Parallel and Distributed Processing

and ApplicationsISPEC Information Security Practice and ExperienceITNG International Conference on Information Technology New GenerationsIWSEC International Workshop on SecurityJAR(J) Journal of Automated ReasoningJCP(J) Journal of ComputersJCS(J) Journal of Computer SecurityJCST(J) Journal of Computer Science and TechnologyJDA(J) Journal of Discrete AlgorithmsJDFP(J) Journal of Digital Forensic PracticeJNCA(J) Journal of Network and Computer ApplicationsJOC(J) Journal of CryptologyJSAC(J) IEEE Journal of Selected Areas in CommunicationsJSS(J) Journal of Systems and SoftwareJUCS(J) The Journal of Universal Computer ScienceKDD Knowledge Discovery and Data MiningKES Knowledge-Based Intelligent Information & Engineering SystemsLCN IEEE Conference on Local Computer NetworksLISA USENIX Large Installation Systems Administration ConferenceMicro(J) IEEE MicroMobiHoc Mobile Ad Hoc Networking and ComputingMobiSys International Conference on Mobile Systems, Applications,

and ServicesMSN Mobile Sensor NetworksMSWiM International Workshop on Modeling Analysis and Simulation of Wireless

and Mobile SystemsMUE Multimedia and Ubiquitous EngineeringNDSI Networked Systems Design and ImplementationNDSS Network and Distributed System Security SymposiumNPC Network and Parallel ComputingNSPW Workshop on New Security Paradigms

14

Acronym Journals/Conferences full nameOOPSLA Conference on Object-Oriented Programming Systems,

Languages, and ApplicationsOSDI Operating Systems Design and ImplementationOTM OTM Conferences /WorkshopsPairing Pairing-Based CryptographyPAISI Pacific Asia Workshop on Intelligence and Security InformaticsPAKDD Pacific-Asia Conference on Knowledge Discovery and Data MiningPerCom IEEE International Conference on Pervasive Computing

and CommunicationsPET Privacy Enhancing TechnologiesPKC Public Key CryptographyPODC ACM SIGACT-SIGOPS Symposium on Principles of

Distributed ComputingPOLICY Policies for Distributed Systems and NetworksPOPL Symposium on Principles of Programming LanguagesPPPJ Principles and Practice of Programming in JavaPSD Privacy in Statistical DatabasesPWC IFIP WG6.8 PublicationsRAID Recent Advances in Intrusion DetectionSAC ACM Symposium on Applied ComputingSACMAT ACM Symposium on Access Control Models and TechnologiesSASN Workshop on Security of ad hoc and Sensor NetworksSCN Security and Cryptography for NetworksSDM Secure Data ManagementSE Software EngineeringSEC Information SecuritySECURWARE International Conference on Emerging Security

Information, Systems and TechnologiesSERA Software Engineering Research and ApplicationsSEUS Software Technologies for Future Embedded and Ubiquitous SystemsSIAM(J) SIAM Journal on ComputingSIGCOMM ACM SIGCOMM ConferenceSIGCSE Technical Symposium on Computer Science EducationSIGFSE Foundations of Software EngineeringSIGMOD(J) SIGMOD RecordSIGMOD ACM SIGMOD ConferenceSOIR(J) Software Process Improvement and PracticeSOUPS Symposium On Usable Privacy and SecuritySP(J) IEEE Security & PrivacySP IEEE Security & PrivacySSS Self-Stabilizing SystemsStorageSS Storage Security And SurvivabilityTALG(J) ACM Transactions on AlgorithmsTC(J) IEEE Transactions on ComputersTCC Theory of CryptographyTCS(J) Theoretical Computer ScienceTDHMS(J) Transactions on Data Hiding and Multimedia SecurityTDSC(J) IEEE Transactions on Dependable and Secure Computing

15

Acronym Journals/Conferences full nameTIFS(J) IEEE Transactions on Information Forensics and SecurityTIP(J) IEEE Transactions on Image ProcessingTISSEC(J) ACM Transactions on Information and System SecurityTJS(J) The Journal of SupercomputingTKDE(J) Transactions on Knowledge and Data EngineeringTMC IEEE Transactions on Mobile ComputingTMM(J) IEEE Transactions on MultimediaTOPLAS(J) ACM Transactions on Programming Languages and SystemsTSE(J) IEEE Transactions on Software EngineeringTSMC(J) IEEE Transactions on Systems, Man, and CyberneticsUIC Ubiquitous Intelligence and ComputingUSENIX USENIX Technical ConferenceVLDB(J) The VLDB JournalVLDB Very Large Data BasesVTC Vehicular Technology ConferenceWCNC IEEE Wireless Communications & Networking ConferenceWINE Workshop on Internet and Network EconomicsWISA Workshop on Information Security ApplicationsWISEC Wireless Network SecurityWPES Workshop on Privacy in the Electronic SocietyWWW International World Wide Web Conferences

16

B. Centrality results

0

5

10

15

20

25

30

0

5

10

15

20

25

CCS

IJSN(J)

ICDM SP

SACM

AT

TISSEC

(J)

ESORICS

JCS(J)

TKDE(J)

ACN

SASIACR

YPT

NDSS

TIFS(J)

SDM

ACSAC

ICICS

TDHMS(J)

EuroVis

TMM(J)

ICPA

DS

TIP(J) DF

IJIS(J)

TDSC(J)

ICIP

PSD

DSP(J)

DIM

VA


0

5

10

15

20

25

30

35

0

5

10

15

20

25

30

35


0

5

10

15

20

25

0

5

10

15

20

25

30


Figure 6: Comparison of high degree nodes. This figure shows the nodes with degree is greater than the average degree in each network.

17

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35


0

0.1

0.2

0.3

0.4

0.5

0.6

0

0.1

0.2

0.3

0.4

0.5

0.6


0

0.1

0.2

0.3

0.4

0.5

0.6

0

0.1

0.2

0.3

0.4

0.5

0.6

CCS

LISA

CGO

ASIACR

YPT

FOCS

JOC(J)

SIGCSE

USENIX

ASPLO

STISSEC

(J)

ESORICS

SACM

AT

SIGMOD

SIGMOD(J)

WPES

DIM

VA CSF

ACSAC

WWW

EDBT

VLDB(J)

SIGFSE

CRYPTO

RAID SP

ICISS


Figure 7: Comparison of high eccentricity nodes. This figure shows the nodes with eccentricity is greater than the average eccentricity in eachnetwork.

18

0

50

100

150

200

250

300

350

400

450

0

200

400

600

800

1000

1200

1400

1600

1800

CCS SACMAT IJSN(J) SP ICDM JCS(J) TISSEC(J) SP(J) ESORICS


0

100

200

300

400

500

600

700

800

900

1000

IEICET(J) ACNS HCI FAW APWeb PAKDD ICCSA

0

200

400

600

800

1000

1200

CCS WPES NDSS


0

50

100

150

200

250

300

ISI ComCom(J) IEICET(J) PerCom

0

100

200

300

400

500

600

700

800

900

1000

CCS WPES ASPLOS USENIX TISSEC(J) CRYPTO ACSAC


Figure 8: Comparison of high betweenness nodes. This figure shows the nodes with betweeness is greater than the average betweeness in eachnetwork.

19

0

0.005

0.01

0.015

0.02

0.025

0

0.001

0.002

0.003

0.004

0.005

0.006

0.007

0.008

0.009

0.01


0

0.005

0.01

0.015

0.02

0.025

0

0.005

0.01

0.015

0.02

0.025

CCS

WPES

NDSS

Pairing

TCC

SOUPS

CRYPTO

FOCS

SIAM(J)

JNCA

(J)

ISCC

TISSEC

(J)

SACM

AT

ACIS

POLICY

ESORICS

ENTC

S(J)

SEC

ColCom

ICISS


0

0.005

0.01

0.015

0.02

0.025

0.03

0

0.002

0.004

0.006

0.008

0.01

0.012

0.014

0.016

0.018

CCS

WPES

ACSAC

CRYPTO

TISSEC

(J)

ASPLO

S

ESORICS

SIGMOD(J)

CSF

SIGMOD

VLDB(J)

EDBT

ASIACR

YPT

JOC(J)

USENIX

DIM

VA LISA

SACM

AT

WWW

SIGFSE

RAID

ICISS


Figure 9: Comparison of high closeness nodes. This figure shows the nodes with closeness is greater than the average closeness in each network.

20

Documents

Methodology for analysis of the research trend: a case ... · PDF fileMethodology for analysis of the research trend: a case ... computer science. Most traditional research ... analysis